Phoury Lei

Using Geotags to Derive Rich Tag-Clouds for Image Annotation

Geotagging has become popular for many multimedia applications. In this chapter, we present an integrated and intuitive system for location-driven tag suggestion, in the form of tag-clouds, for geotagged photos. Potential tags from multiple sources are extracted and weighted. Sources include points of interest (POI) tags from a public Geographic Names Information System (GNIS) database, community tags from Flickr® pictures, and personal tags shared through users’ own, family, and friends’ photo collections. To increase the effectiveness of GNIS POI tags, bags of place-name tags are first retrieved, clustered, and then re-ranked using a combined tf-idf and spatial distance criteria. The community tags from photos taken in the vicinity of the input geotagged photo are ranked according to distance and visual similarity to the input photo. Personal tags from other personally related photos inherently carry a significant weight due more to their high relevance than to both the generic place-name tags and community tags, and are ranked by weights that decay over time and distance differences. Finally, a rich set of the most relevant location-driven tags is presented to the user in the form of individual tag clouds under the three mentioned source categories. The tag clouds act as intuitive suggestions for tagging an input image. We also discuss quantitative and qualitative findings from a user study that we conducted. Evaluation has revealed the respective benefits of the three categories toward the effectiveness of the integrated tag suggestion system.

A consumer video search system by audio-visual concept classification

Indexing and searching the massive amount of consumer videos in the open domain is increasingly important. Due to the lack of text descriptions as well as the difficulties in analyzing the content of consumer videos, little work has been conducted to provide video search engines in the consumer domain. In this paper, we develop a content-based consumer video search system based on multi-modal concept classification. The system supports the query-by-example access mechanism, by exploiting the query-by-concept search paradigm underneath, where online concept classification is conducted over the query video by integrating both visual and audio information. The system adopts an audio-visual grouplet representation that captures salient audio-visual signatures to describe the video content for efficient concept classification. Experiments over the large-scale Columbia Consumer Video set show the effectiveness of the developed system.

Dynamic media show drivable by semantics

We demonstrate a system to generate dynamic media shows that are significantly richer than static slide shows, which are currently the most popular form of photo playback. The goal is to enable media reliving experiences that are aesthetically pleasing, interactive, and semantically drivable as they center on people, locations, time, and events discovered in a media collection. Dynamic shows allow for better sharing of ones media collections in diverse social networks because people have different time availabilities and perspectives, and hence may want to interact, customize, and reroute the media flow per their individual needs.