Geoff Barnbrook

Language Independent Statistical Software for Corpus Exploration

In this report two programs for statistical analysis of concordance lines are described. The programs have been developed for analyzing he lexical context of a given word. It is shown how different parameter settings influence the outcome of collocational analysis, and how the concept of collocation can be extended to allow the extraction of lines typical for a word from a set of concordance lines. Even though all the examples are for English, the software is completely language independent and only requires minimal linguistic resources.

The concept of collocation

The use of the word collocation has varied a great deal since it was first borrowed into English around the sixteenth century. The history of these changes is covered in detail as part of this chapter, but since there is still considerable variation in its use as a technical linguistic term it might be helpful to establish how the word is used in this book. Generally the word is used in three main ways: to describe the way in which words group together in their normal use in texts to describe the analysis tool used to explore this grouping and to assess its significance and implications and, more controversially, to describe an aspect of language production in which pre-fabricated chunks of language are used to build up utterances. To appreciate these concepts of collocation properly and to understand their importance in modern linguistics, it may be useful to get an overview of the development of the term and the ways in which it has been used.

Pedagogy, translation and natural language processing

The production of pedagogic materials for the teaching of English as a foreign language has also been influenced by the need to give adequate guidance on collocation. In Chapter 1 we have already seen that an understanding of learners’ needs led to a greater awareness of the significance of collocation in the early twentieth century, again preceding the formal recognition of its place in linguistic theory. Pedagogy, including the specialised lexicography that accompanies it, is another major current application of collocation, explored in detail below in section 6.2.

Concordances and lexicography

The pervasive nature of collocation in authentic language, forming as it does a crucial part of native speaker authenticity, gives it a special importance in language description and analysis and language teaching. This part of the book explores some of the ways in which it has been incorporated into and used within these areas.

Collocation and language theory: twentieth century

The collocation dictionaries described in section 1.11 of Chapter 1 were compiled using techniques similar to those used in the production of more conventional dictionaries. OCD makes specific reference to the use of the British National Corpus (OCD, viii) both for the collocations them-selves and for the usage examples, while BBI only specifically refers to the sources of new entries for the revised edition (BBI, vii–viii) — mainly reviews and suggestions from readers of the first edition. The ways in which the lexicographers of collocation obtain and authenticate their data have important implications for the perceived role of collocation as an aspect of language.

Collocation and language theory: recent developments

Sections 2.1 to 2.3 above discussed the impact that collocation has had on modern linguistic theory. In particular, towards the end of section 2.3 it was suggested that any move away from the open-choice model of language towards the idiom principle in the interpretation of texts has significant implications for the basis of the production of texts. This part of the book focuses specifically on these implications and on the areas of linguistic theory most affected.