Joseph P. Levy

Extracting semantic representations from word co-occurrence statistics: a computational study.

The idea that at least some aspects of word meaning can be induced from patterns of word co-occurrence is becoming increasingly popular. However, there is less agreement about the precise computations involved, and the appropriate tests to distinguish between the various possibilities. It is important that the effect of the relevant design choices and parameter values are understood if psychological models using these methods are to be reliably evaluated and compared. In this article, we present a systematic exploration of the principal computational possibilities for formulating and validating representations of word meanings from word co-occurrence statistics. We find that, once we have identified the best procedures, a very simple approach is surprisingly successful and robust over a range of psychologically relevant evaluation measures.

Extracting semantic representations from word co-occurrence statistics: stop-lists, stemming, and SVD.

In a previous article, we presented a systematic computational study of the extraction of semantic representations from the word–word co-occurrence statistics of large text corpora. The conclusion was that semantic vectors of pointwise mutual information values from very small co-occurrence windows, together with a cosine distance measure, consistently resulted in the best representations across a range of psychologically relevant semantic tasks. This article extends that study by investigating the use of three further factors—namely, the application of stop-lists, word stemming, and dimensionality reduction using singular value decomposition (SVD)—that have been used to provide improved performance elsewhere. It also introduces an additional semantic task and explores the advantages of using a much larger corpus. This leads to the discovery and analysis of improved SVD-based methods for generating semantic representations (that provide new state-of-the-art performance on a standard TOEFL task) and the identification and discussion of problems and misleading results that can arise without a full systematic study.

Extracting Semantic Representations from Large Text Corpora

Many connectionist language processing models have now reached a level of detail at which more realistic representations of semantics are required. In this paper we discuss the extraction of semantic representations from the word co-occurrence statistics of large text corpora and present a preliminary investigation into the validation and optimisation of such representations. We find that there is significantly more variation across the extraction procedures and evaluation criteria than is commonly assumed.

Learning Lexical Properties from Word Usage Patterns: Which Context Words Should be Used?

Several recent papers have described how lexical properties of words can be captured by simple measurements of which other words tend to occur close to them. At a practical level, word co–occurrence statistics are used to generate high dimensional vector space representations and appropriate distance metrics are defined on those spaces. The resulting co–occurrence vectors have been used to account for phenomena ranging from semantic priming to vocabulary acquisition. We have developed a simple and highly efficient system for computing useful word co–occurrence statistics, along with a number of criteria for optimizing and validating the resulting representations. Other workers have advocated various methods for reducing the number of dimensions in the co–occurrence vectors. LundB LandauerD and Lowe&McDonald [8] have used a statistical reliability criterion. We have used a simpler framework that orders and truncates the dimensions according to their word frequency. Here we compare how the different methods perform for two evaluation criteria and briefly discuss the consequences of the different methodologies for work within cognitive or neural computation.

Inclined to see it your way: Do altercentric intrusion effects in visual perspective taking reflect an intrinsically social process?

It has been suggested that some aspects of mental state understanding recruit a rudimentary, but fast and efficient, processing system, demonstrated by the obligatory slowing down of judgements about what the self can see when this is incongruent with what another can see. We tested the social nature of this system by investigating to what extent these altercentric intrusions are elicited under conditions that differed in their social relevance and, further, how these related to self-reported social perspective taking and empathy. In Experiment 1, adult participants were asked to make “self” or “other” perspective-taking judgements during congruent (“self” and “other” can see the same items) or incongruent conditions (“self” and “other” cannot see the same items) in conditions that were social (i.e., involving a social agent), semisocial (an arrow), or nonsocial (a dual-coloured block). Reaction time indices of altercentric intrusion effects were present across all conditions, but were significantly stronger for the social than for the less social conditions. Self-reported perspective taking and empathy correlated with altercentric intrusion effects in the social condition only. In Experiment 2, the significant correlations for the social condition were replicated, but this time with gaze duration indices of altercentric intrusion effects. Findings are discussed with regard to the degree to which this rudimentary system is socially specialized and how it is linked to more conceptual understanding.

Explorations in the derivation of word co-occurrence statistics

Recent work has demonstrated that counts of which other words co-occur with a word of interest can reflect interesting properties of that word. We have studied aspects of this kind of methodology by systematically examining the effects of different combinations of parameters used in the preparation of co-occurrence statistics. Several psychologically relevant evaluation measures are used. We have found that successful performance on the evaluation tasks depends on the correct selection of parameters such as window size and distance metric.

Sex and individual differences in induced and evoked EEG measures of action observation

We used two established methods for analysing the EEG response of the neurotypical adult human brain to examine the execution and observation of simple motor actions. In one, execution or observation of a button-press in response to a tone caused a decrease in the power at 8-13 Hz (mu) frequencies. In the other, the response preparation (or the inferred response preparation when these actions are observed in another person) was measured by the averaged response time-locked potentials measured over motor cortex--the readiness potential. Results indicated that the mirrored readiness potentials were bilaterally generated. We found sex differences for both measures. However, whereas females showed a greater degree of response for the mu power measure during the observation of movement only, males showed larger readiness potentials during both movement performance and observation. Both measures have been claimed to be neural correlates of mirror systems in the brain where processes responsible for actions are linked to the perception of such actions. Such mirror systems have also been implicated in higher order social cognition such as empathy. However, we found no correlations between either of our EEG measures and self-report scales of social cognition. The results imply sex differences in the measured systems and for mirroring that are not directly related to social cognition. We suggest that the results may indicate two dissociable motor mirroring systems that can be measured by induced and evoked EEG.

Limiting Factors for Mapping Corpus-Based Semantic Representations to Brain Activity

To help understand how semantic information is represented in the human brain, a number of previous studies have explored how a linear mapping from corpus derived semantic representations to corresponding patterns of fMRI brain activations can be learned. They have demonstrated that such a mapping for concrete nouns is able to predict brain activations with accuracy levels significantly above chance, but the more recent elaborations have achieved relatively little performance improvement over the original study. In fact, the absolute accuracies of all these models are still currently rather limited, and it is not clear which aspects of the approach need improving in order to achieve performance levels that might lead to better accounts of human capabilities. This paper presents a systematic series of computational experiments designed to identify the limiting factors of the approach. Two distinct series of artificial brain activation vectors with varying levels of noise are introduced to characterize how the brain activation data restricts performance, and improved corpus based semantic vectors are developed to determine how the word set and model inputs affect the results. These experiments lead to the conclusion that the current state-of-the-art input semantic representations are already operating nearly perfectly (at least for non-ambiguous concrete nouns), and that it is primarily the quality of the fMRI data that is limiting what can be achieved with this approach. The results allow the study to end with empirically informed suggestions about the best directions for future research in this area.

Theory of mind in emerging reading comprehension: a longitudinal study of early indirect and direct effects

The relation between childrens theory of mind (ToM) and emerging reading comprehension was investigated in a longitudinal study over 2.5years. A total of 80 children were tested for ToM, decoding, language skills, and executive function (EF) at Time 1 (mean age=3;10 [years;months]). At Time 2 (mean age=6;03), childrens word reading efficiency, language skills, and reading comprehension were measured. Mediation analysis showed that ToM at Time 1, when children were around 4years old, indirectly predicted Time 2 reading comprehension, when children were 6years old, via language ability after controlling for age, nonverbal ability, decoding, EF, and earlier language ability. Importantly, ToM at 4years also directly predicted reading comprehension 2.5years later at 6years. This is the first longitudinal study to show a direct contribution of ToM to reading comprehension in typical development. Findings are discussed in terms of the simple view of reading (SVR); ToM not only supports reading comprehension indirectly by facilitating language but also contributes to it directly over and above the SVR. The potential role of metacognition is considered when accounting for the direct contribution of early ToM to later reading comprehension.

USING ENRICHED SEMANTIC REPRESENTATIONS IN PREDICTIONS OF HUMAN BRAIN ACTIVITY

There have been many different theoretical proposals for ways of representing word meaning in a distributed fashion. We ourselves have put forward a framework for expressing aspects of lexical semantics in terms of patterns of word co-occurrences measured in large linguistic corpora. Recent advances in the modelling of fMRI measures of brain activity have started to examine patterns of activation across the cortex rather than averaging activity across a sub-volume. Mitchell et al. [11] have shown that simple linear models can successfully predict fMRI data from patterns of word co-occurrence for a task where participants mentally generate properties for presented word-picture pairs. Using their MRI data, we replicate their models and extend them to use our independently optimised co-occurrence patterns to demonstrate that enriched representations of word/concept meaning produce significantly better predictions of brain activity. We also explore several aspects of the parameter space underlying the supervised learning techniques used in these models.