Alex D. Hwang

Semantic guidance of eye movements during real-world scene inspection

Semantic guidance of eye movements during real-world scene inspection Alex D. Hwang (ahwang@cs.umb.edu) Hsueh-Cheng Wang (hcwang@cs.umb.edu) Marc Pomplun (marc@cs.umb.edu) Department of Computer Science, University of Massachusetts Boston 100 Morrissey Blvd., Boston, MA 02125-3393, USA Abstract This is the first study to measure semantic guidance during scene inspection, based on the efforts by two other research groups, namely the development of the LabelMe object- annotated image database and the LSA@CU text/word latent semantic analysis tool, which computes the conceptual distance between two terms. Our analysis reveals the existence of semantic guidance during scene inspection, that is, eye movements during scene inspection being guided by a semantic factor reflecting the conceptual relation between the currently fixated object and the target object of the following saccade. This guidance may facilitate memorization of the scene for later recall by viewing semantically related objects consecutively. Keywords: semantic guidance; contextual guidance; eye tracking; eye movement, scene inspection. Introduction Real-world scenes are filled with objects representing not only visual information, but also meanings and semantic relations with other objects in the scene. The guidance of eye movements based on visual appearance (low-level visual features) has been well studied in terms of both bottom-up (e.g., Bruce & Tsotsos, 2006; Henderson, 2003; Itti & Koch, 2001; Parkhurst, Law & Niebur, 2002) and top- down control of visual attention (e.g., Henderson, Brockmole, Castelhano & Mack, 2007; Hwang, Higgins & Pomplun, 2009; Peters & Itti, 2007; Pomplun, 2006; Zelinsky, 2008; Zelinsky, Zhang, Yu, Chen & Samaras, 2006) as well as neurological aspects (e.g., Corbetta & Shulman, 2002; Egner, Monti, Trittschuh, Wienecke, Hirsch & Mesulam, 2008). Although there has been research on high-level contextual effects on visual search using global features (e.g., Neider & Zelinski, 2006; Torralba, Oliva, Castelhano & Henderson, 2006) and primitive semantic effects based on co- occurrence of objects in term of implicit learning (e.g., Chun & Jiang, 1998; Chun & Phelps, 1999; Manginelli & Pollmann, 2008), effects on eye movements by object meaning and object relations, Semantic guidance, have not been studied because of a few hurdles that make such study more complicated: (1) Object segmentation is difficult, (2) semantic relations among objects are hard to define, and (3) a quantitative measure of semantic guidance has to be developed. Automated segmentation of images and labeling is one of the crucial steps for further understanding of image context, and there have been numerous attempts to solve this problem, ranging from global classification of scenes to individual region labeling (Athanasiadis, Mylonas, Avrithis, & Kollias, (2007); Boutell, Luo, Shena & Brown, 2004; Le Saux, & Amato, 2004; Luo & Savakis, 2001), but results were rather disappointing compared to human performance. Thanks to the LabelMe object-annotated image database (Russell, Torralba, Murphy & Freeman, 2008) developed by the MIT Computer Science and Artificial Intelligence Laboratory (CSAIL), various scenes with annotated objects are available to the public, which helps to bypass the first hurdle. In order to convincingly compute semantic or conceptual relations between objects, the co-occurrence of objects has to be analyzed in a large number of scenes images. Unfortunately, collecting and analyzing a sufficient amount of annotated scenes is unfeasible with the currently available data sources. Since semantic relations are formed at the conceptual rather than at the visual level, relations do not have to be derived from image databases. Any database that can generate a collection of contexts or knowledge can be used to represent the semantic meaning of objects. A useful mathematical method for such representation for computer modeling and simulation is Latent Semantic Analysis (LSA), which is based on the analysis of representative corpora of natural text. It transforms the occurrence matrix from large corpora into a relation between the terms/concepts, and a relation between those concepts and the documents (Landauer, Foltz & Laham, 1998). Since annotated data in LabelMe are text descriptions of objects, their semantic or conceptual relation can be processed with LSA. In this study, the LSA@CU text/word latent semantic analysis tool is used to pass the second hurdle. Equipped with above tools, we computed a series of semantic salience maps for each labeled object in a subject’s visual scan path. These salience maps approximated the transition probabilities for the following saccade to the other labeled objects in the scene, assuming that eye movements were entirely guided by the semantic relations between objects. Under this assumption, the probability of a gaze transition between two objects is proportionate to the strength of their semantic relation. Subsequently, the amount of semantic guidance was measured by the Receiver Operator Characteristic (ROC), which computes the extent

Conspicuity Of Object Features Determines Local Versus Global Mental Rotation Strategies

Mental rotation is a top down process that is evidenced by the time required todiscriminate between identical and mirrored objects increasing linearly with theangular deviation between them. Although mental rotation is a distributedprocessingtask,itsdependenceon objectfeaturesis notwellunderstood(Nakatani& Pollatsek, 2004). Therefore we investigated the effect of congruent versusincongruent color markers on mental rotation strategies. It is unclear just how theavailability of colorin images modulatesmentalrotation strategies for visualobjectrecognition(Tanakaet al.,2001).We tracked participants’eyemovementsto getaninsightintothesestrategies.Theresultssuggestthatfor highlydemandingrotationtasks, distinctive features induce multiple local comparisons of object structurewhereas the absence of such features tends to induce mental rotation of largerpartsof theobjector theentireobject.