Javier Insa-Cabrera

Comparing humans and AI agents

Comparing humans and machines is one important source of information about both machine and human strengths and limitations. Most of these comparisons and competitions are performed in rather specific tasks such as calculus, speech recognition, translation, games, etc. The information conveyed by these experiments is limited, since it portrays that machines are much better than humans at some domains and worse at others. In fact, CAPTCHAs exploit this fact. However, there have only been a few proposals of general intelligence tests in the last two decades, and, to our knowledge, just a couple of implementations and evaluations. In this paper, we implement one of the most recent test proposals, devise an interface for humans and use it to compare the intelligence of humans and Q-learning, a popular reinforcement learning algorithm. The results are highly informative in many ways, raising many questions on the use of a (universal) distribution of environments, on the role of measuring knowledge acquisition, and other issues, such as speed, duration of the test, scalability, etc.

Evaluating a reinforcement learning algorithm with a general intelligence test

In this paper we apply the recent notion of anytime universal intelligence tests to the evaluation of a popular reinforcement learning algorithm, Q-learning. We show that a general approach to intelligence evaluation of AI algorithms is feasible. This top-down (theory-derived) approach is based on a generation of environments under a Solomonoff universal distribution instead of using a pre-defined set of specific tasks, such as mazes, problem repositories, etc. This first application of a general intelligence test to a reinforcement learning algorithm brings us to the issue of task-specific vs. general AI agents. This, in turn, suggests new avenues for AI agent evaluation and AI competitions, and also conveys some further insights about the performance of specific algorithms.

Factors of collective intelligence: How smart are agent collectives?

The dynamics and characteristics behind intelligent cognitive systems lie at the heart of understanding, and devising, successful solutions to a variety of multiagent problems. Despite the extant literature on collective intelligence, important questions like “how does the effectiveness of a collective compare to its isolated members?” and “are there some general rules or properties shaping the spread of intelligence across various cognitive systems and environments?” remain somewhat of a mystery. In this paper we develop the idea of collective intelligence by giving some insight into a range of factors hindering and influencing the effectiveness of interactive cognitive systems. We measure the influence of each examined factor on intelligence independently, and empirically show that collective intelligence is a function of them all simultaneously. We further investigate how the organisational structure of equally sized groups shapes their effectiveness. The outcome is fundamental to the understanding and prediction of the collective performance of multiagent systems, and for quantifying the emergence of intelligence over different environmental settings.

Instrumental Properties of Social Testbeds

The evaluation of an ability or skill happens in some kind of testbed, and so does with social intelligence. Of course, not all testbeds are suitable for this matter. But, how can we be sure of their appropriateness? In this paper we identify the components that should be considered in order to measure social intelligence, and provide some instrumental properties in order to assess the suitability of a testbed.