Luís Filipe Teófilo

Adapting Strategies to Opponent Models in Incomplete Information Games: A Reinforcement Learning Approach for Poker

Researching into the incomplete information games (IIG) field requires the development of strategies which focus on optimizing the decision making process, as there is no unequivocal best choice for a particular play. As such, this paper describes the development process and testing of an agent able to compete against human players on Poker --- one of the most popular IIG. The used methodology combines pre-defined opponent models with a reinforcement learning approach. The decision-making algorithm creates a different strategy against each type of opponent by identifying the opponents type and adjusting the rewards of the actions of the corresponding strategy. The opponent models are simple classifications used by Poker experts. Thus, each strategy is constantly adapted throughout the games, continuously improving the agents performance. In light of this, two agents with the same structure but different rewarding conditions were developed and tested against other agents and each other. The test results indicated that after a training phase the developed strategy is capable of outperforming basic/intermediate playing strategies thus validating this approach.

High-Level Language to Build Poker Agents

On the last decade Poker has been one of the most interesting subjects for artificial intelligence, because it is a game that requires game playing agents to deal with an incomplete information and stochastic scenario. The development of Poker agents has seen significant advances but it is still hard to evaluate agents’ performance against human players. This is either because it is illicit to use agents in online games, or because human players cannot create agents that play like themselves due to lack of knowledge on computer science and/or AI. The purpose of this work is to fill the gap between poker players and AI in Poker by allowing players without programming skills to build their own agents. To meet this goal, a high-level language of poker concepts – PokerLang – was created, whose structure is easy to read and interpret for domain experts. This language allows for the quick definition of an agent strategy. A graphical application was also created to support the writing of PokerLang strategies. To validate this approach, some Poker players created their agents using the graphical application. Results validated the usability of the application and the language that supports it. Moreover, the created agents showed very good results against agents developed by other experts.

A multi-layered segmentation method for nucleus detection in highly clustered microscopy imaging: A practical application and validation using human U2OS cytoplasm---nucleus translocation images

Fluorescent microscopy imaging is a popular and well-established method for biomedical research. However, the large number of images created in each research trial quickly eliminates the possibility of a manual annotation; thus, the need for automatic image annotation is quickly becoming an urgent need. Furthermore, the high clustering indexes and noise observed in these images contribute to a complex issue, which has attracted the attention of the scientific community. In this paper, we present a fully automated method for annotating fluorescent confocal microscopy images in highly complex conditions. The proposed method relies on a multi-layered segmentation and declustering process, which begins with an adaptive segmentation step using a two-level Otsu’s Method. The second layer is comprised of two probabilistic classifiers, responsible for determining how many components may constitute each segmented region. The first of these employs rule-based reasoning grounded on the decreasing harmonic pattern observed in the region area density function, while the second one consists of a Support Vector Machine trained with features derived from the log likelihood ratio function of Gaussian mixture models of each region. Our results indicate that the proposed method is able to perform the identification and annotation process on par with an expert human subject, thus presenting itself a viable alternative to the traditional manual approach.

Simulation and Performance Assessment of Poker Agents

The challenge in developing agents for incomplete information games resides in the fact that the maximum utility decision for given information set is not always ascertainable. For large games like Poker, the agents’ strategies require opponent modeling, since Nash equilibrium strategies are hard to compute. In light of this, simulation systems are indispensable for accurate assessment of agents’ capabilities. Nevertheless, current systems do not accommodate the needs of computer poker research since they were designed mainly as an interface for human players competing against agents. In order to contribute towards improving computer poker research, a new simulation system was developed. This system introduces scientifically unexplored game modes with the purpose of providing a more realistic simulation environment, where the agent must play carefully to manage its initial resources. An evolutionary simulation feature was also included so as to provide support for the improvement of adaptive strategies. The simulator has built-in odds calculation, an agent development API, other platform agents and several variants support and an agent classifier with realistic game indicators including exploitability estimation. Tests and qualitative analysis have proven this simulator to be faster and better suited for thorough agent development and performance assessment.

Multi-modal natural interaction in game design: a comparative analysis of player experience in a large scale role-playing game

Previous work on player experience research has focused on identifying the major factors involving content creation and interaction. This has encouraged a large investment in new types of physical interaction artefacts (e.g. Wiimote™, Rock Band™, Kinect™). However, these artefacts still require custom interaction schemes to be developed for them, which critically limits the number of commercial videogames and multimedia applications that can benefit from those. Moreover, there is currently no agreement as to which factors better describe the impact that natural and complex multi-modal user interaction schemes have on users’ experiences—a gap in part created by the limitations in adapting this type of interaction to existing software. Thus, this paper presents a generic middleware framework for multi-modal natural interfaces which enables game-independent data acquisition that encourages further advancement on this domain. Furthermore, our framework can then redefine the interaction scheme of any software tool by mapping body poses and voice commands to traditional input means (keyboard and mouse). We have focused on digital games, where the use of physical interaction artefacts has become mainstream. The validation methods for this tool consisted of a series of increasing difficulty stress tests, with a total of 25 participants. Also, a pilot study was conducted on a further 16 subjects which demonstrated mainly positive impact of natural interfaces on player’s experience. The results supporting this were acquired when subjects played a complex commercial role-playing game whose mechanics were adapted using our framework; statistical tests on the obtained Fun ratings, along with subjective participant opinions indicate that this kind of natural interaction indeed has a significant impact on player’s experience and enjoyment. However, different impact patterns emerge from this analysis, which seem to fit with standing theories of player experience and immersion.

GEMINI: A Generic Multi-Modal Natural Interface Framework for Videogames

In recent years videogame companies have recognized the role of player engagement as a major factor in user experience and enjoyment. This encouraged a greater investment in new types of game controllers such as the WiiMoteTM, Rock BandTM instruments and the KinectTM. However, the native software of these controllers was not originally designed to be used in other game applications. This work addresses this issue by building a middleware framework, which maps body poses or voice commands to actions in any game. This not only warrants a more natural and customized user-experience but it also defines an interoperable virtual controller. In this version of the framework, body poses and voice commands are respectively recognized through the Kinect’s built-in cameras and microphones. The acquired data is then translated into the native interaction scheme in real time using a lightweight method based on spatial restrictions. The system is also prepared to use Nintendo’s WiimoteTM as an auxiliary and unobtrusive gamepad for physically or verbally impractical commands. System validation was performed by analyzing the performance of certain tasks and examining user reports. Both confirmed this approach as a practical and alluring alternative to the game’s native interaction scheme. In sum, this framework provides a game-controlling tool that is totally customizable and very flexible, thus expanding the market of game consumers.

Automatic analysis of leishmania infected microscopy images via gaussian mixture models

This work addresses the issue of automatic organic component detection and segmentation in confocal microscopy images. The proposed method performs cellular/parasitic identification through adaptive segmentation using a two-level Otsus Method. Segmented regions are divided using a rule-based classifier modeled on a decreasing harmonic function and a Support Vector Machine trained with features extracted from several Gaussian mixture models of the segmented regions. Results indicate the proposed method is able to count cells and parasites with accuracies above 90%, as well as perform individual cell/parasite detection in multiple nucleic regions with approximately 85% accuracy. Runtime measures indicate the proposed method is also adequate for real-time usage.

A Probabilistic Approach to Organic Component Detection in Leishmania Infected Microscopy Images

This paper proposes a fully automated method for annotating confocal microscopy images, through organic component detection and segmentation. The organic component detection is performed through adaptive segmentation using a two-level Otsu’s Method. Two probabilistic classifiers then analyze the detected regions, as to how many components may constitute each one. The first of these employs rule-based reasoning centered on the decreasing harmonic patterns observed in the region area density functions. The second one consists of a Support Vector Machine trained with features derived from the log likelihood ratios of incrementally Gaussian mixture modeling detected regions. The final step pairs the identified cellular and parasitic components, computing the standard infection ratios on biomedical research. Results indicate the proposed method is able to perform the identification and annotation processes on par with expert human subjects, constituting a viable alternative to the traditional manual approach.

A Poker Game Description Language

During the last decade, Computer Poker has become the preferred test-bed for validating developments on the extensive-form game and multi-agent systems research domains. Because Poker is a game with hundreds of variants differing from each other by their betting structure, number of cards in the deck or winning conditions, numerous agents have been created for several different variants of the game. However, there is not a single unified description model that allows for those agents to be tested across different Poker variants inexpensively. For this reason, we introduce the Poker Game Description Language (PGDL), which, unlike other incomplete information GDLs, is uniquely focused on Poker agent development and testing. PGDL is integrated into a playable system which not only makes available a basic Agent Development API in Prolog, but also provides a simple in-built agent which can adapt to user-defined rules. In addition, this framework has a simple GUI which both basic and advanced test subjects demonstrated to be adequate and easy-to-use when defining new PGDL instances. We believe that despite the existence of more generic general game playing systems, the fact that our language natively supplies a shared infrastructure, common to all Poker variants, renders our approach very pertinent for Poker agent development. Tests demonstrated that our language was capable of describing the most popular Poker variants.

Rule based strategies for large extensive-form games: A specification language for No-Limit Texas Hold’em agents

Poker is used to measure progresses in extensive-form games research due to its unique characteristics: it is a game where playing agents have to deal with incomplete information and stochastic scenarios and a large number of decision points. The development of Poker agents has seen significant advances in one-on-one matches but there are still no consistent results in multiplayer and in games against human experts. In order to allow for experts to aid the improvement of the agents’ performance, we have created a high-level strategy specification language. To support strategy definition, we have also developed an intuitive graphical tool. Additionally, we have also created a strategy inferring system, based on a dynamically weighted Euclidean distance. This approach was validated through the creation of simple agents and by successfully inferring strategies from 10 human players. The created agents were able to beat previously developed mid-level agents by a good profit margin.