Ole-Christoffer Granmo

Solving two‐armed Bernoulli bandit problems using a Bayesian learning automaton

Purpose – The two‐armed Bernoulli bandit (TABB) problem is a classical optimization problem where an agent sequentially pulls one of two arms attached to a gambling machine, with each pull resulting either in a reward or a penalty. The reward probabilities of each arm are unknown, and thus one must balance between exploiting existing knowledge about the arms, and obtaining new information. The purpose of this paper is to report research into a completely new family of solution schemes for the TABB problem: the Bayesian learning automaton (BLA) family.Design/methodology/approach – Although computationally intractable in many cases, Bayesian methods provide a standard for optimal decision making. BLA avoids the problem of computational intractability by not explicitly performing the Bayesian computations. Rather, it is based upon merely counting rewards/penalties, combined with random sampling from a pair of twin Beta distributions. This is intuitively appealing since the Bayesian conjugate prior for a bino...

Solving Stochastic Nonlinear Resource Allocation Problems Using a Hierarchy of Twofold Resource Allocation Automata

In a multitude of real-world situations, resources must be allocated based on incomplete and noisy information. However, in many cases, incomplete and noisy information render traditional resource allocation techniques ineffective. The decentralized Learning Automata Knapsack Game (LAKG) was recently proposed for solving one such class of problems, namely the class of Stochastic Nonlinear Fractional Knapsack Problems. Empirically, the LAKG was shown to yield a superior performance when compared to methods which are based on traditional parameter estimation schemes. This paper presents a completely new online Learning Automata (LA) system, namely the Hierarchy of Twofold Resource Allocation Automata (H-TRAA). In terms of contributions, we first of all, note that the primitive component of the H-TRAA is a Twofold Resource Allocation Automaton (TRAA) which possesses novelty in the field of LA. Second, the paper contains a formal analysis of the TRAA, including a rigorous proof for its convergence. Third, the paper proves the convergence of the H-TRAA itself. Finally, we demonstrate empirically that the H-TRAA provides orders of magnitude faster convergence compared to the LAKG for simulated data pertaining to two-material unit-value functions. Indeed, in contrast to the LAKG, the H-TRAA scales sublinearly. Consequently, we believe that the H-TRAA opens avenues for handling demanding real-world applications such as the allocation of sampling resources in large-scale Web accessibility assessment problems. We are currently working on applying the H-TRAA solution to the web-polling and sample-size detection problems applicable to the world wide web.

Routing Bandwidth-Guaranteed Paths in MPLS Traffic Engineering: A Multiple Race Track Learning Approach

This paper presents an efficient adaptive online routing algorithm for the computation of bandwidth-guaranteed paths in multiprotocol label switching witching (MPLS)-based networks by using a learning scheme that computes an optimal ordering of routes. The contribution of this work is twofold. The first is that we propose a new class of solutions other than those available in the literature, incorporating the family of stochastic random races (RR) algorithms. The most popular previously proposed MPLS-based traffic engineering (TE) solutions attempt to find a superior path to route an incoming setup request. Our algorithm, on the other hand, tries to learn an optimal ordering of the paths through which requests can be routed according to the rank of the paths in the order learned by the algorithm. The second contribution of our work is that we have proposed a routing algorithm that has a performance superior to the important algorithms in the literature. Our conclusions are based on three important performance criteria: 1) the rejection ratio, 2) the percentage of accepted bandwidth, and 3) the average route computation time per request. Although some of the previously proposed algorithms were designed to achieve low rejection and high throughput of route requests, they are unreasonably slow. Our algorithm, on the other hand, in general attempts to reject the least number of requests, achieves the highest throughput, and computes routes in the fastest possible time when compared to the algorithms that we used as benchmarks for comparison.

Supporting timeliness and accuracy in distributed real-time content-based video analysis

Real-time content-based access to live video data requires content analysis applications that are able to process the video data at least as fast as the video data is made available to the application and with an acceptable error rate. Statements as this express quality of service (QoS) requirements to the application. In order to provide some level of control of the QoS provided, the video content analysis application must be scalable and resource aware so that requirements of timeliness and accuracy can be met by allocating additional processing resources.In this paper we present a general architecture of video content analysis applications including a model for specifying requirements of timeliness and accuracy. The salient features of the architecture include its combination of probabilistic knowledge-based media content analysis with QoS and distributed resource management to handle QoS requirements, and its independent scalability at multiple logical levels of distribution. We also present experimental results with an algorithm for QoS-aware selection of configurations of feature extractor and classification algorithms that can be used to balance requirements of timeliness and accuracy against available processing resources. Experiments with an implementation of a real-time motion vector based object-tracking application, demonstrate the scalability of the architecture.

Service selection in stochastic environments: a learning-automaton based solution

In this paper, we propose a novel solution to the problem of identifying services of high quality. The reported solutions to this problem have, in one way or the other, resorted to using so-called “Reputation Systems” (RSs). Although these systems can offer generic recommendations by aggregating user-provided opinions about the quality of the services under consideration, they are, understandably, prone to “ballot stuffing” and “badmouthing” in a competitive marketplace. In general, unfair ratings may degrade the trustworthiness of RSs, and additionally, changes in the quality of service, over time, can render previous ratings unreliable. As opposed to the reported solutions, in this paper, we propose to solve the problem using tools provided by Learning Automata (LA), which have proven properties capable of learning the optimal action when operating in unknown stochastic environments. Furthermore, they combine rapid and accurate convergence with low computational complexity. In addition to its computational simplicity, unlike most reported approaches, our scheme does not require prior knowledge of the degree of any of the above mentioned problems associated with RSs. Instead, it gradually learns the identity and characteristics of the users which provide fair ratings, and of those who provide unfair ratings, even when these are a consequence of them making unintentional mistakes.Comprehensive empirical results show that our LA-based scheme efficiently handles any degree of unfair ratings (as long as these ratings are binary—the extension to non-binary ratings is “trivial”, if we use the S-model of LA computations instead of the P-model). Furthermore, if the quality of services and/or the trustworthiness of the users change, our scheme is able to robustly track such changes over time. Finally, the scheme is ideal for decentralized processing. Accordingly, we believe that our LA-based scheme forms a promising basis for improving the performance of RSs in general.

A novel strategy for solving the stochastic point location problem using a hierarchical searching scheme.

Stochastic point location (SPL) deals with the problem of a learning mechanism (LM) determining the optimal point on the line when the only input it receives are stochastic signals about the direction in which it should move. One can differentiate the SPL from the traditional class of optimization problems by the fact that the former considers the case where the directional information, for example, as inferred from an Oracle (which possibly computes the derivatives), suffices to achieve the optimization-without actually explicitly computing any derivatives. The SPL can be described in terms of a LM (algorithm) attempting to locate a point on a line. The LM interacts with a random environment which essentially informs it, possibly erroneously, if the unknown parameter is on the left or the right of a given point. Given a current estimate of the optimal solution, all the reported solutions to this problem effectively move along the line to yield updated estimates which are in the neighborhood of the current solution.1 This paper proposes a dramatically distinct strategy, namely, that of partitioning the line in a hierarchical tree-like manner, and of moving to relatively distant points, as characterized by those along the path of the tree. We are thus attempting to merge the rich fields of stochastic optimization and data structures. Indeed, as in the original discretized solution to the SPL, in one sense, our solution utilizes the concept of discretization and operates a uni-dimensional controlled random walk (RW) in the discretized space, to locate the unknown parameter. However, by moving to nonneighbor points in the space, our newly proposed hierarchical stochastic searching on the line (HSSL) solution performs such a controlled RW on the discretized space structured on a superimposed binary tree. We demonstrate that the HSSL solution is orders of magnitude faster than the original SPL solution proposed by Oommen. By a rigorous analysis, the HSSL is shown to be optimal if the effectiveness (or credibility) of the environment, given by p, is greater than the golden ratio conjugate. The solution has been both analytically solved and simulated, and the results obtained are extremely fascinating, as this is the first reported use of time reversibility in the analysis of stochastic learning. The learning automata extensions of the scheme are currently being investigated.

Crowd Models for Emergency Evacuation: A Review Targeting Human-Centered Sensing

Emergency evacuation of crowds is a fascinating phenomenon that has attracted researchers from various fields. Better understanding of this class of crowd behavior opens up for improving evacuation policies and smarter design of buildings, increasing safety. Recently, a new class of disruptive technology has appeared: Human-centered sensing which allows crowd behavior to be monitored in real-time, and provides the basis for real-time crowd control. The question then becomes: to what degree can previous crowd models incorporate this development, and what areas need further research? In this paper, we provide a survey that describes some widely used crowd models and discuss their advantages and shortages from the angle of human-centered sensing. Our review reveals important research opportunities that may contribute to an improved and more robust emergency management.

Accelerated Bayesian learning for decentralized two-armed bandit based decision making with applications to the Goore Game

The two-armed bandit problem is a classical optimization problem where a decision maker sequentially pulls one of two arms attached to a gambling machine, with each pull resulting in a random reward. The reward distributions are unknown, and thus, one must balance between exploiting existing knowledge about the arms, and obtaining new information. Bandit problems are particularly fascinating because a large class of real world problems, including routing, Quality of Service (QoS) control, game playing, and resource allocation, can be solved in a decentralized manner when modeled as a system of interacting gambling machines.Although computationally intractable in many cases, Bayesian methods provide a standard for optimal decision making. This paper proposes a novel scheme for decentralized decision making based on the Goore Game in which each decision maker is inherently Bayesian in nature, yet avoids computational intractability by relying simply on updating the hyper parameters of sibling conjugate priors, and on random sampling from these posteriors. We further report theoretical results on the variance of the random rewards experienced by each individual decision maker. Based on these theoretical results, each decision maker is able to accelerate its own learning by taking advantage of the increasingly more reliable feedback that is obtained as exploration gradually turns into exploitation in bandit problem based learning.Extensive experiments, involving QoS control in simulated wireless sensor networks, demonstrate that the accelerated learning allows us to combine the benefits of conservative learning, which is high accuracy, with the benefits of hurried learning, which is fast convergence. In this manner, our scheme outperforms recently proposed Goore Game solution schemes, where one has to trade off accuracy with speed. As an additional benefit, performance also becomes more stable. We thus believe that our methodology opens avenues for improved performance in a number of applications of bandit based decentralized decision making.

Learning-Automaton-Based Online Discovery and Tracking of Spatiotemporal Event Patterns

Discovering and tracking of spatiotemporal patterns in noisy sequences of events are difficult tasks that have become increasingly pertinent due to recent advances in ubiquitous computing, such as community-based social networking applications. The core activities for applications of this class include the sharing and notification of events, and the importance and usefulness of these functionalities increase as event sharing expands into larger areas of ones life. Ironically, instead of being helpful, an excessive number of event notifications can quickly render the functionality of event sharing to be obtrusive. Indeed, any notification of events that provides redundant information to the application/user can be seen to be an unnecessary distraction. In this paper, we introduce a new scheme for discovering and tracking noisy spatiotemporal event patterns, with the purpose of suppressing reoccurring patterns, while discerning novel events. Our scheme is based on maintaining a collection of hypotheses, each one conjecturing a specific spatiotemporal event pattern. A dedicated learning automaton (LA)-the spatiotemporal pattern LA (STPLA)-is associated with each hypothesis. By processing events as they unfold, we attempt to infer the correctness of each hypothesis through a real-time guided random walk. Consequently, the scheme that we present is computationally efficient, with a minimal memory footprint. Furthermore, it is ergodic, allowing adaptation. Empirical results involving extensive simulations demonstrate the superior convergence and adaptation speed of STPLA, as well as an ability to operate successfully with noise, including both the erroneous inclusion and omission of events. An empirical comparison study was performed and confirms the superiority of our scheme compared to a similar state-of-the-art approach. In particular, the robustness of the STPLA to inclusion as well as to omission noise constitutes a unique property compared to other related approaches. In addition, the results included, which involve the so-called “ presence sharing” application, are both promising and, in our opinion, impressive. It is thus our opinion that the proposed STPLA scheme is, in general, ideal for improving the usefulness of event notification and sharing systems, since it is capable of significantly, robustly, and adaptively suppressing redundant information.

Using Stochastic AI Techniques to Achieve Unbounded Resolution in Finite Player Goore Games and its Applications

The Goore Game (GG) introduced by M. L. Tsetlin in 1973 has the fascinating property that it can be resolved in a completely distributed manner with no intercommunication between the players. The game has recently found applications in many domains, including the field of sensor networks and quality-of-service (QoS) routing. In actual implementations of the solution, the players are typically replaced by learning automata (LA). The problem with the existing reported approaches is that the accuracy of the solution achieved is intricately related to the number of players participating in the game -which, in turn, determines the resolution. In other words, an arbitrary accuracy can be obtained only if the game has an infinite number of players. In this paper, we show how we can attain an unbounded accuracy for the GG by utilizing no more than three stochastic learning machines, and by recursively pruning the solution space to guarantee that the retained domain contains the solution to the game with a probability as close to unity as desired. The paper also conjectures on how the solution can be applied to some of the application domains