Harald Benzing

The Sequential Design of Bernoulli Experiments Including Switching Costs

We consider a sequence of N trials, each of which must be performed on one of two given Bernoulli experiments. We assume the success probability of one experiment is known and the other is unknown. Performing two successive trials on different experiments incurs a switching cost. The problem is to choose an experiment for each trial in order to maximize the expected number of successes minus the expected switching costs. We show that an optimal design shares some well-known monotonicity properties, such as the “stopping-rule” and the “stay-on-a-winner” rule. We also show how to use these results to derive a simple algorithm for calculating the optimal design. Since the model contains the one-armed bandit problem as a special case, we also obtain new proofs for known results.

On monotone optimal decision rules and the stay-on-a-winner rule for the two-armed bandit

SummaryConsider the following optimization problem: Find a decision rule δ such thatw(x, δ (x))=maxaw(x, a) for allx under the constraint δ (x)∈D (x). We give conditions for the existence of monotone optimal decision rules δ. The term ‘monotone’ is used in a general sense. The well-known stay-on-a-winner rules for the two-armed bandit can be characterized as monotone decision rules by including the stage number intox and using a special ordering onx. This enables us to give simple conditions for the existence of optimal rules that are stay-on-a-winner rules. We extend results ofBerry andKalin/Theodorescu to the case of dependent arms.

On the k-armed Bernoulli bandit: monotonicity of the total reward under an arbitrary prior distribution

We investigate monotonicity properties of the success probabilities and the total reward when the number of previously observed successes and failures change. Using a well-known Bayesian approach and dynamic programming we give conditions in terms of the covariances of the posterior distributions and in terms of the support of the prior distribution. Special order relations for the number of successes and failures allow a simple and unified treatment of different cases. The results extend some of the investigations of Hengartner/Kalin/Theodorescu [1].

Structured policies for a sequential design problem with general distributions

Consider the problem of allocating two treatments in a sequence of N trials such that the total expected reward is maximized. We assume that the distribution of outcomes is known for one of the treatments only (so-called one-armed bandit problem).In a suitable dynamic programming model we examine the structural properties of an optimal policy. In particular we extend results known for Bernoulli distributed outcomes to more general distributions.Our main technical results are monotonicity and convexity of the total expected reward.

On a sequential two-action decision model 1 with unbounded reward functions

Several decision problems such as bandit problems, stopping problems, and portfolio problems, can be considered as special sequential two-action Markov decision models. In this paper such models are studied; a stopping rule is given, and monotonicity properties of the maximum expected total discounted reward and of an optimal policy are established. The theory is illustrated by two examples.

On the Bernoulli three-armed bandit problem 1

The paper is concerned with the Bernoulli three-armed bandit problem with finite horizon and with one known success probability (arm) and success probabilities for the two remaining unknown arms distributed according to an arbitrary prior distribution (Bayesian approach). A discounted dynamic programming model is used for the study of this problem, Monotonicity of the optimal expected cumulative discounted reward, a stopping rule, and optimal policies are investigated.

Monotone optimal decision rules and their computation

For a given objective functionw(x, a) onX × A, a maximizinga=δ(x) has to be determined for eachx in the totally ordered setX. We give conditions onw such that there is a monotone δ which can be computed recursively ifA is finite.

Bounds for the approximation of dynamic programs

SummaryWe consider a general finite stage dynamic programming model. Bounds are derived for the approximation of the minimum expected total cost and of the optimal policy. The theory is applied to an inventory model to give bounds for “good” order policies.ZusammenfassungEs wird ein allgemeines dynamisches Optimierungsmodell mit endlichem Horizont betrachtet. Für verschiedene Näherungsverfahren für die minimalen erwarteten Gesamtkosten und die optimale Politik werden Schranken angegeben. Die Theorie wird sodann auf ein Lagerhaltungsmodell angewandt, um Schranken für „gute“ Bestellpolitiken zu erhalten.

Der Zweiarmige Bandit mit Abhängigen Armen und die Stay-on-a-Winner Eigenschaft

Wir betrachten das folgende Problem der optimalen Versuchsplanung: Es seien zwei BERNOULLI-Experimente, genannt Experiment 1 bzw. Experiment 2 mit unbekannten Erfolgswahrscheinlichkeiten T1 bzw. T2 gegeben. In einer Serie von N Versuchen soll fur jeden Versuch ein Experiment so ausgewahlt werden, das die erwartete Anzahl von Erfolgen maximal wird.

Zur Berechnung Monotoner Minimisatoren

In der (sequentiellen) Entscheidungstheorie stosst man haufig auf das folgende Optimierungsproblem: Zu jedem x ∈ X (Stichprobe oder Zustand) bestimme man δ(x) ∈ A (Entscheidung), so dass eine gegebene Zielfunktion w(x,a) fur a = δ(x) minimiert wird. Die Entscheidungsfunktion δ heisst dann Minimisator von w. X und A seien endliche Mengen. Die praktische Berechnung des Minimisators δ vereinfacht sich erheblich, wenn man aus Eigenschaften von w ablesen kann, dass δ monoton ist. δ lasst sich dann charakterisieren durch die Punkte ya ∈ X, a ∈ A, an den δ vom Wert a-1 auf den Wert a springt (mehrere benachbarte Werte ya konnen identisch sein).