Kay L. O'Halloran

Classroom Discourse in Mathematics: A Multisemiotic Analysis

The impact of the multisemiotic nature of mathematics on classroom discourse is examined from a systemic functional linguistic perspective. Mathematical discourse is multisemiotic because it involves the use of the semiotic resources of mathematical symbolism, visual display and language. There is constant movement between the three resources as the primary code and with shifts between spoken and written modes in classrooms. The complexity of pedagogical discourse arises because each of the three semiotic resources have their own unique lexicogrammatical systems for encoding meaning, and these interact to shape the nature of constructions found in the classroom. This is especially significant in the case of mathematical symbolism where meaning is encoded unambiguously in the most economical manner possible through specific grammatical strategies, one of which involves the use of multiple levels of rankshifted configurations of mathematical Operative processes and participants. The dense texture of mathematics pedagogical discourse arises from the inclusion of these symbolic constructions in the linguistic metadiscourse. The complex tracking and reference patterns arising from the reconfigurations of symbolic Operative processes and participants in the board text and oral discourse are also examined. Lastly, the notion of semiotic metaphor whereby shifts in meaning of functional element occur and new entities are introduced with movements between semiotic codes is discussed.

Systemic functional-multimodal discourse analysis (SF-MDA): constructing ideational meaning using language and visual imagery

The systemic functional (SF) approach to multimodal discourse analysis (MDA) is concerned with the theory and practice of analysing meaning arising from the use of multiple semiotic resources in discourses which range from written, printed and electronic texts to material lived-in reality. The SF-MDA approach developed in this article explores the meaning arising through the use of language and visual imagery in printed texts. This involves investigation of linguistic and visual forms of semiosis, and formulation of cross-functional systems such as colour. An integrative platform based on the SF metafunctional principle is proposed, and intersemiotic mechanisms and systems (content and expression strata) are developed to capture the expansion of meaning which occurs when linguistic and visual forms combine. The SF-MDA approach is demonstrated through the analysis of ideational meaning in a print advertisement. The practical approach involves the use of digital technology in the form of image-editing software which gives rise to a more detailed semantic and ideological interpretation. The analysis reveals how metaphorical constructions of meaning (i.e. semiotic metaphors) take place across linguistic and visual elements.

Intersemiotic Texture: analyzing cohesive devices between language and images

Recent research on multimodal discourse has explored the nature of semantic relations between different semiotic resources. Drawing on the interpretation of language as a social semiotic resource, this article proposes Intersemiotic Texture as the crucial property of coherent multimodal texts and presents a preliminary framework for cohesive devices between language and images. The framework is illustrated through examination of print media to demonstrate how the image–text relations are meta-functionally orchestrated across experiential, textual and logical meanings at the discourse stratum. A discourse-based model is suggested to analyze image–text logical relations complementary to existing grammar-based approaches. This research also develops a meta-language to describe Intersemiotic Cohesive Devices from two complementary perspectives: Intersemiotic Cohesion not only functions to integrate different modes together when multimodal discourse is conceptualized as a finished product, it also constitutes essential text-forming resources for semantic expansions across language and images during the ongoing contextualization of meanings in real time.

Interdependence, interaction and metaphor in multisemiotic texts

Systemic frameworks for discourse analysis of visual/verbal and mathematics texts are used to examine the functionality of language? visual display and mathematical symbolism, and the meaning arising from interaction and interdependence between these semiotic codes in joint constructions. The nature of the interaction between options, classified as unmarked or marked, determines the meaning made in that instance and, more generally, provides an impetus for the expansion of the meaning potential of each semiotic. The phenomenon where semantic shifts occur when functional elements are reconstrued in another semiotic is called semiotic metaphor. Rather than the notion of an overlay of meaning typically associated with grammatical metaphor, the potential exists here for more dramatic forms of semantic shift. Examples of semiotic metaphor categorised as ‘parallel semiotic metaphor’ and ‘divergent semiotic metaphor’ are given in the text analysis.

Spatial pedagogy: mapping meanings in the use of classroom space

The positioning and movement of the teacher in the classroom are fundamental to the pedagogical process. Specific spaces in the classroom take on certain meanings because of the nature of pedagogic discourse that occurs on the site and the positioning and distance of the site relative to the students and the teaching resources. Spatial pedagogy is realised through the patterns of positioning and the directionality of movement, as well as the intersemiotic correspondences in the use of space with other semiotic resources (e.g. language, gesture and teaching materials). This paper investigates the different types of space in the classroom and their associated meanings. It also discusses an approach for annotating the teacher’s use of space, and the usefulness of visualising this annotation through digital graphical methods. The notion of ‘structured informality’ in the classroom is proposed through data analysis of two teachers conducting similar lessons, but with apparently different pedagogical styles.

‘Toward a global knowledge enterprise’: university websites as portals to the ongoing marketization of higher education

Using a critical ‘hypermodal approach’ informed by social semiotics, this paper investigates the changing discourses of marketization found on the website of the National University of Singapore over a 14-year period. Analysis of visual-spatial features and action potentials of progressive versions of the site reveals changes in the website functions, first from providing information about resources and expertise to addressing potential students as consumers of goods and of products offered by the university. Later we find the website pointing not so much to education as a process of learning and mentoring but as a type of lifestyle, experience, and abstracted personal transformation and journey. Here the university positions itself increasingly in a global as opposed to national community, where students are to be fine-tuned to the new kinds of marketized demands this will bring.

“David Byrne really does love PowerPoint”: art as research on semiotics and semiotic technology

This article presents a reading of David Byrnes Envisioning Emotional Epistemological Information, an art work created with MicroSofts presentation software PowerPoint, as an instance of creative research on semiotics and semiotic technology. It reveals commonalities and differences between Byrnes ideas about PowerPoint and related ideas from linguistics and semiotics, and is intended as a contribution to research on PowerPoint, and on semiotic technologies generally, as well as to efforts aimed at developing criteria for evaluating art as research on semiotics.

Learning human photo shooting patterns from large-scale community photo collections

Social photo sharing platforms on the Internet (e.g. Flickr) host billions of publicly accessible photos captured by millions of individual users from all over the world. These user-contributed and geo-tagged photo collections provide insights into human sociocultural life and provide important clues for understanding people’s engagement and reaction to places and events around the world today. In this paper, we analyze over 2 million geo-tagged images uploaded by 12,000 individual Flickr users to investigate the photograph shooting patterns of different user groups; that is, tourist and local, Asian and European, and male and female users. Specifically, we make use of visual features extracted on single monocular images and their spatial configurations to infer 3D depth information of the photographs to establish the preferred shooting scale (close-up or far-distant) of the user groups. The results reveal which objects and scenes interest different groups of people and how these preferences change over space and time. As such, the research offers a new approach to the human sciences which study the individual, groups and society.

The multimodal representation of emotion in film : integrating cognitive and semiotic approaches

Abstract This study provides a semiotic theorization of how emotion is represented in film to complement the cognitive approach, which focuses on how film elicits emotion from viewers. Drawing upon social semiotic theories and cognitive theories of emotion, we develop a multimodal framework in which filmic representation of emotion is seen as combinations of semiotic choices derived from cognitive components of emotion. The semiotic model is employed to investigate how emotive meaning is realized through verbal and nonverbal resources. At the discursive level of film, the choices available in the shot organization of eliciting condition and expression are examined. The paper demonstrates how the social semiotic approach, combined with cognitive theory of emotion structure, is able to provide a comprehensive theoretical account of how various film techniques represent emotion. It is also significant for the study of viewer emotion, which to a large degree stems from character emotion.

Sequential Visual Discourse Frames

The cover of the weekly American news magazine Time (8 May 2006) features ‘The lives and ideas of the world’s most influential people’ with a photograph montage of the people included on the 2006 Time list. Preceding the cover story ‘People Who Shape Our World’, Time Asia contains 1–2 full-page colour advertisements for Rolex, Shell, DHL, Breitling, Mercedes-Benz, Toshiba, Lufthansa, Sony Ericsson, Lenovo, Longines and Rado products and services, in addition to some news articles. The cover story (72 pages) contains further advertisements for Bayers Healthcare, Asian Games, Thai Airways, Toyota Formula 1 and Cartier which unfold generically until something unusual happens. The Cartier advertisement (Frame 1, Illustration 9.1) becomes a two-page visual spread (Frames 2–3, Illustration 9.1) that unfolds at the centre to reveal the four-page article Time 100 ‘Power Couples’ (Centrefold, Illustration 9.1). Upon folding back these two pages (Frames 2–3) and turning the page, the Cartier advertisement contains one further page (Frame 4, Illustration 9.1).