William S. Lovejoy

A survey of algorithmic methods for partially observed Markov decision processes

A partially observed Markov decision process (POMDP) is a generalization of a Markov decision process that allows for incomplete information regarding the state of the system. The significant applied potential for such processes remains largely unrealized, due to an historical lack of tractable solution methodologies. This paper reviews some of the current algorithmic alternatives for solving discrete-time, finite POMDPs over both finite and infinite horizons. The major impediment to exact solution is that, even with a finite set of internal system states, the set of possible information states is uncountably infinite. Finite algorithms are theoretically available for exact solution of the finite horizon problem, but these are computationally intractable for even modest-sized problems. Several approximation methodologies are reviewed that have the potential to generate computationally feasible, high precision solutions.

Computationally feasible bounds for partially observed Markov decision processes

A partially observed Markov decision process (POMDP) is a sequential decision problem where information concerning parameters of interest is incomplete, and possible actions include sampling, surveying, or otherwise collecting additional information. Such problems can theoretically be solved as dynamic programs, but the relevant state space is infinite, which inhibits algorithmic solution. This paper explains how to approximate the state space by a finite grid of points, and use that grid to construct upper and lower value function bounds, generate approximate nonstationary and stationary policies, and bound the value loss relative to optimal for using these policies in the decision problem. A numerical example illustrates the methodology.

Integrated Product Design for Marketability and Manufacturing

Using as a point of departure previous work in marketing on optimal concept selection that utilizes product attribute-based customer preference and product cost models, the authors consider the con...

Some monotonicity results for partially observed Markov decision processes

This paper provides sufficient conditions for the optimal value in a discrete-time, finite, partially observed Markov decision process to be monotone on the space of state probability vectors ordered by likelihood ratios. The paper also presents sufficient conditions for the optimal policy to be monotone in a simple machine replacement problem, and, in the general case, for the optimal policy to be bounded from below by an easily calculated monotone function.

Hospital Operating Room Capacity Expansion

A large midwestern hospital is expecting an increase in surgical caseload. New operating room (OR) capacity can be had by building new ORs or extending the working hours in the current ORs. The choice among these options is complicated by the fact that patients, surgeons and surgical staff, and hospital administrators are all important stakeholders in the health service operation, and each has different priorities. This paper investigates the trade-offs among three performance criteria (wait to get on schedule, scheduled procedure start-time reliability, and hospital profits), which are of particular importance to the different constituencies. The objective is to determine how the hospital can best expand its capacity, acknowledging the key role that each constituency plays in that objective. En route, the paper presents supporting analysis for process improvements and suggestions for optimal participation-inducing staff contracts for extending OR hours of operation.

Five Decades of Operations Management and the Prospects Ahead

Operations and Supply Chains is the current title for a department that has evolved through several different titles in recent years, reflecting its evolving mission from a focus on classical operations research at the time of ORSAs founding 50 years ago toward an embrace of a broader body of theory. Throughout this evolution, the focus on applied problems and the goal of improving practice through the development of suitable theory has remained constant The Operations and Supply Chains Department promotes the theory underlying the practice of operations management, which encompasses the design and management of the transformation processes in manufacturing and service organizations that create value for society. Operations is the function that is uniquely associated with the design and management of these processes. The problem domains of concern to the department have been, and remain, the marshalling of inputs, the transformation itself, and the distribution of outputs in pursuit of this value-creating end. Over the past 50 years the department has had a variety of titles, reflecting an evolving understanding of the boundaries of the operations function. In this article we celebrate past accomplishments, identify current challenges, and anticipate a future that is as exciting and opportunity-rich as any our field has seen.

Comparison of Some Suboptimal Control Policies in Medical Drug Therapy

In drug therapy, efficient dosage policies are needed to maintain drug concentrations at target. The relationship between the concentration of a drug and the dosages is often described by compartment models in which the parameters are unknown, although prior knowledge may be available and can be updated after blood samples are taken during the therapy. In this paper we define some tractable policies for adaptive control of drug concentrations in compartment models and compare their performances using computer simulation in a one-compartment model. We also discuss the effects of assuming normal priors, discrete approximation of a continuous prior, using nonquadratic costs, and information probing. From the simulation we derive intuition as to what types of policies perform well and address the topic of actively versus passively learning.

Technical Note-On the Convexity of Policy Regions in Partially Observed Systems

We offer a simple observation that unifies many of the existing convex policy region results for partially observed systems.

Bargaining Chains

We consider a firm that designs a new product and wishes to bring it to market but does not have ownership or control over all of the resources required to make that happen. The firm must select and contract with one of several possible tier 1 suppliers for necessary inputs, who do the same with their (tier 2) suppliers, etc. This general situation is common in industry. We assume tier-wise negotiations, sole sourcing within each tier, complete local information, and horizontal competition. We develop a bargaining-based solution to the negotiations between two adjacent multifirm tiers and show its consistency with familiar solution concepts from the theories of bargaining and cooperative games. We then link up multiple bargaining modules to generate chainwide predictions for efficiency and profitability in supply chains with an arbitrary number of tiers and an arbitrary number of firms per tier. We investigate the implications of the results for investments in process improvements or supplier development.

Efficient Structures for Innovative Social Networks

What lines of communication among members of an organization are most productive in the early, ideation phase of innovation? We investigate this question with a recombination and selection model of knowledge transfer operating through a social network. We find that ideation is accelerated when people in the organization dynamically churn through a large (ideally the entire population) set of conversational partners over time, which naturally begets short path lengths and eliminates information bottlenecks. Group meetings, in which the content of conversations is available to all for consideration, are another way to learn in parallel and accelerate the ideation process, although for complex problems they may not offer significant advantages over the best decentralized networks. The idealized core-periphery graphs emerge as an important family on the time--cost efficient frontier. New sociometrics for the analyses of innovation processes emerge from this investigation.