David Sweeney

Efficient and precise interactive hand tracking through joint, continuous optimization of pose and correspondences

Fully articulated hand tracking promises to enable fundamentally new interactions with virtual and augmented worlds, but the limited accuracy and efficiency of current systems has prevented widespread adoption. Todays dominant paradigm uses machine learning for initialization and recovery followed by iterative model-fitting optimization to achieve a detailed pose fit. We follow this paradigm, but make several changes to the model-fitting, namely using: (1) a more discriminative objective function; (2) a smooth-surface model that provides gradients for non-linear optimization; and (3) joint optimization over both the model pose and the correspondences between observed data points and the model surface. While each of these changes may actually increase the cost per fitting iteration, we find a compensating decrease in the number of iterations. Further, the wide basin of convergence means that fewer starting points are needed for successful model fitting. Our system runs in real-time on CPU only, which frees up the commonly over-burdened GPU for experience designers. The hand tracker is efficient enough to run on low-power devices such as tablets. We can track up to several meters from the camera to provide a large working volume for interaction, even using the noisy data from current-generation depth cameras. Quantitative assessments on standard datasets show that the new approach exceeds the state of the art in accuracy. Qualitative results take the form of live recordings of a range of interactive experiences enabled by this new approach.

Learning to be a depth camera for close-range human capture and interaction

We present a machine learning technique for estimating absolute, per-pixel depth using any conventional monocular 2D camera, with minor hardware modifications. Our approach targets close-range human capture and interaction where dense 3D estimation of hands and faces is desired. We use hybrid classification-regression forests to learn how to map from near infrared intensity images to absolute, metric depth in real-time. We demonstrate a variety of human-computer interaction and capture scenarios. Experiments show an accuracy that outperforms a conventional light fall-off baseline, and is comparable to high-quality consumer depth cameras, but with a dramatically reduced cost, power consumption, and form-factor.

Data and life on the street

What does the abundance of data and proliferation of data-making methods mean for the ordinary person, the person on the street? And, what could they come to mean? In this paper, we present an overview of a year-long project to examine just such questions and complicate, in some ways, what it is to ask them. The project is a collective exercise in which we – a mixture of social scientists, designers and makers – and those living and working on one street in Cambridge (UK), Tenison Road, are working to think through how data might be materialised and come to matter. The project aims to better understand the specificities and contingencies that arise when data is produced and used in place. Mid-way through the project, we use this commentary to give some background to the work and detail one or two of the troubles we have encountered in putting locally relevant data to work. We also touch on a methodological standpoint we are working our way into and through, one that we hope complicates the separations between subject and object in data-making and opens up possibilities for a generative refiguring of the manifold relations.

Exploring the Design Space for Energy-Harvesting Situated Displays

We explore the design space of energy-neutral situated displays, which give physical presence to digital information. We investigate three central dimensions: energy sources, display technologies, and wireless communications. Based on the power implications from our analysis, we present a thin, wireless, photovoltaic-powered display that is quick and easy to deploy and capable of indefinite operation in indoor lighting conditions. The display uses a low-resolution e-paper architecture, which is 35 times more energy-efficient than smaller-sized high-resolution displays. We present a detailed analysis on power consumption, photovoltaic energy harvesting performance, and a detailed comparison to other display-driving architectures. Depending on the ambient lighting, the display can trigger an update every 1 -- 25 minutes and communicate to a PC or smartphone via Bluetooth Low-Energy.

Displays as a Material: A Route to Making Displays More Pervasive

For digital displays to become more pervasive, they need to have the properties of materials like textiles and plastic film. Moving away from the row-column addressing architecture that dominates traditional displays, the authors propose an architecture that relies on autonomous pixels - that is, pixels that independently sense input and convert it to a corresponding visual output. Two different prototypes reveal the challenges and potential of autonomous pixels, highlighting how digital displays supplied as a flexible material can help foster the development of various new applications. This department is part of a special issue on pervasive displays.

Surfacing Small Worlds through Data-In-Place

We present findings from a five-week deployment of voting technologies in a city neighbourhood. Drawing on Marres’ (2012) work on material participation and Massey’s (2005) conceptualisation of space as dynamic, we designed the deployment such that the technologies (which were situated in residents’ homes, on the street, and available online) would work in concert, cutting across the neighbourhood to make visible, juxtapose and draw together the different ‘small worlds’ within it. We demonstrate how the material infrastructure of the voting devices set in motion particular processes and interpretations of participation, putting data in place in a way that had ramifications for the recognition of heterogeneity. We conclude that redistributing participation means not only opening up access, so that everyone can participate, or even providing a multitude of voting channels, so that people can participate in different ways. Rather, it means making visible multiplicity, challenging notions of similarity, and showing how difference may be productive.