Simon P. King

From context to content: leveraging context to infer media metadata

The recent popularity of mobile camera phones allows for new opportunities to gather important metadata at the point of capture. This paper describes a method for generating metadata for photos using spatial, temporal, and social context. We describe a system we implemented for inferring location information for pictures taken with camera phones and its performance evaluation. We propose that leveraging contextual metadata at the point of capture can address the problems of the semantic and sensory gaps. In particular, combining and sharing spatial, temporal, and social contextual metadata from a given user and across users allows us to make inferences about media content.

Over-exposed?: privacy patterns and considerations in online and mobile photo sharing

As sharing personal media online becomes easier and widely spread, new privacy concerns emerge - especially when the persistent nature of the media and associated context reveals details about the physical and social context in which the media items were created. In a first-of-its-kind study, we use context-aware camerephone devices to examine privacy decisions in mobile and online photo sharing. Through data analysis on a corpus of privacy decisions and associated context data from a real-world system, we identify relationships between location of photo capture and photo privacy settings. Our data analysis leads to further questions which we investigate through a set of interviews with 15 users. The interviews reveal common themes in privacy considerations: security, social disclosure, identity and convenience. Finally, we highlight several implications and opportunities for design of media sharing applications, including using past privacy patterns to prevent oversights and errors.

MMM2: mobile media metadata for media sharing

Cameraphones are rapidly becoming a global platform for everyday digital imaging especially for networked sharing of media from mobile devices. However, their constrained user interfaces and the current network and application infrastructure encumber the basic tasks of transferring, finding, and sharing captured media. We have deployed a prototype context-aware cameraphone application for mobile media sharing (MMM2) that aims to overcome these difficulties. MMM2 leverages the point of capture and of sharing to gather metadata, and uses metadata to support sharing. Based on the early results of the first 6 weeks of a six-month trial involving 60 users, indications are that with MMM2 users are actively capturing and sharing photos. The ability to automatically upload photos from a cameraphone to a web-based photo management application and to automatically suggest sharing recipients at the time of capture based on Bluetooth-sensed co-presence and sharing frequency promise to reduce the current difficulty of mobile media sharing.

Towards context-aware face recognition

In this paper, we focus on the use of context-aware, collaborative filtering, machine-learning techniques that leverage automatically sensed and inferred contextual metadata together with computer vision analysis of image content to make accurate predictions about the human subjects depicted in cameraphone photos. We apply Sparse-Factor Analysis (SFA) to both the contextual metadata gathered in the MMM2 system and the results of PCA (Principal Components Analysis) of the photo content to achieve a 60% face recognition accuracy of people depicted in our cameraphone photos, which is 40% better than media analysis alone. In short, we use context-aware media analysis to solve the face recognition problem for cameraphone photos.

Using Context and Similarity for Face and Location Identification

This paper describes a new approach to the automatic detection of human faces and places depicted in photographs taken on cameraphones. Cameraphones offer a unique opportunity to pursue new approaches to media analysis and management: namely to combine the analysis of automatically gathered contextual metadata with media content analysis to fundamentally improve image content recognition and retrieval. Current approaches to content-based image analysis are not sufficient to enable retrieval of cameraphone photos by high-level semantic concepts, such as who is in the photo or what the photo is actually depicting. In this paper, new methods for determining image similarity are combined with analysis of automatically acquired contextual metadata to substantially improve the performance of face and place recognition algorithms. For faces, we apply Sparse-Factor Analysis (SFA) to both the automatically captured contextual metadata and the results of PCA (Principal Components Analysis) of the photo content to achieve a 60% face recognition accuracy of people depicted in our database of photos, which is 40% better than media analysis alone. For location, grouping visually similar photos using a model of Cognitive Visual Attention (CVA) in conjunction with contextual metadata analysis yields a significant improvement over color histogram and CVA methods alone. We achieve an improvement in location retrieval precision from 30% precision for color histogram and CVA image analysis, to 55% precision using contextual metadata alone, to 67% precision achieved by combining contextual metadata with CVA image analysis. The combination of context and content analysis produces results that can indicate the faces and places depicted in cameraphone photos significantly better than image analysis or context analysis alone. We believe these results indicate the possibilities of a new context-aware paradigm for image analysis.

MMM2: mobile media metadata for photo sharing

Though cameraphones are rapidly becoming the dominant platform for consumer digital photography, users still face difficulties in transferring, managing, and sharing photos captured with cameraphones. The Mobile Media Metadata 2 (MMM2) system removes the difficulty in transferring photos from the device by providing an automatic upload capability and uses metadata about the context in which a photo was captured to simplify photo management and streamline the sharing process. In our MMM2 system, we have leveraged collaborative filtering techniques to infer the likely sharing recipients for photos based on contextual metadata, which allows the system to accurately guess likely share recipients for a photo and present them to the photographer at the time of capture.

Zurfer: mobile multimedia access in spatial, social and topical context

What happens when you can access all the worlds media, but the access is constrained by screen size, bandwidth, attention, and battery life? We present a novel mobile context-aware software prototype that enables access to images on the go. Our prototype utilizes the channel metaphor to give users contextual access to media of interest according to key dimensions: spatial, social, and topical. Our experimental prototype attempts to be playful and simple to use, yet provide powerful and comprehensive media access. A temporally-driven sorting scheme for media items allows quick and easy access to items of interest in any dimension. For ad-hoc tasks, we extend the application with keyword search to deliver the long tail of media and images. Elements of social interaction and communication around the photographs are built into the mobile application, to increase user engagement. The application utilizes Flickr.com as an image and social-network data source, but could easily be extended to support other websites and media formats.

PhotoRouter: destination-centric mobile media messaging

The number of people using cameraphones is growing by tens of millions every month. Yet the majority of cameraphone users have difficulty transferring photos off their phone and sharing them with others. PhotoRouter is a software application for cameraphones that makes the photo sharing process destination-centric by allowing users to focus on who the photo should go to, not how it needs to get there. Attempting to produce an application which meets user needs better than current, technology-centric cameraphone photo sharing applications, we designed PhotoRouter. In this paper we describe PhotoRouters user interface innovations that we will show in our technical demonstration.