John P. Collomosse

A performance evaluation of gradient field HOG descriptor for sketch based image retrieval

We present an image retrieval system for the interactive search of photo collections using free-hand sketches depicting shape. We describe Gradient Field HOG (GF-HOG); an adapted form of the HOG descriptor suitable for Sketch Based Image Retrieval (SBIR). We incorporate GF-HOG into a Bag of Visual Words (BoVW) retrieval framework, and demonstrate how this combination may be harnessed both for robust SBIR, and for localizing sketched objects within an image. We evaluate over a large Flickr sourced dataset comprising 33 shape categories, using queries from 10 non-expert sketchers. We compare GF-HOG against state-of-the-art descriptors with common distance measures and language models for image retrieval, and explore how affine deformation of the sketch impacts search performance. GF-HOG is shown to consistently outperform retrieval versus SIFT, multi-resolution HOG, Self Similarity, Shape Context and Structure Tensor. Further, we incorporate semantic keywords into our GF-HOG system to enable the use of annotated sketches for image search. A novel graph-based measure of semantic similarity is proposed and two applications explored: semantic sketch based image retrieval and a semantic photo montage.

State of the "Art”: A Taxonomy of Artistic Stylization Techniques for Images and Video

This paper surveys the field of nonphotorealistic rendering (NPR), focusing on techniques for transforming 2D input (images and video) into artistically stylized renderings. We first present a taxonomy of the 2D NPR algorithms developed over the past two decades, structured according to the design characteristics and behavior of each technique. We then describe a chronology of development from the semiautomatic paint systems of the early nineties, through to the automated painterly rendering systems of the late nineties driven by image gradient analysis. Two complementary trends in the NPR literature are then addressed, with reference to our taxonomy. First, the fusion of higher level computer vision and NPR, illustrating the trends toward scene analysis to drive artistic abstraction and diversity of style. Second, the evolution of local processing approaches toward edge-aware filtering for real-time stylization of images and video. The survey then concludes with a discussion of open challenges for 2D NPR identified in recent NPR symposia, including topics such as user and aesthetic evaluation.

Gradient field descriptor for sketch based retrieval and localization

We present an image retrieval system driven by free-hand sketched queries depicting shape. We introduce Gradient Field HoG (GF-HOG) as a depiction invariant image descriptor, encapsulating local spatial structure in the sketch and facilitating efficient codebook based retrieval. We show improved retrieval accuracy over 3 leading descriptors (Self Similarity, SIFT, HoG) across two datasets (Flickr160, ETHZ extended objects), and explain how GF-HOG can be combined with RANSAC to localize sketched objects within relevant images. We also demonstrate a prototype sketch driven photo montage application based on our system.

TouchCut: Fast image and video segmentation using single-touch interaction

We present TouchCut; a robust and efficient algorithm for segmenting image and video sequences with minimal user interaction. Our algorithm requires only a single finger touch to identify the object of interest in the image or first frame of video. Our approach is based on a level set framework, with an appearance model fusing edge, region texture and geometric information sampled local to the touched point. We first present our image segmentation solution, then extend this framework to progressive (per-frame) video segmentation, encouraging temporal coherence by incorporating motion estimation and a shape prior learned from previous frames. This new approach to visual object cut-out provides a practical solution for image and video segmentation on compact touch screen devices, facilitating spatially localized media manipulation. We describe such a case study, enabling users to selectively stylize video objects to create a hand-painted effect. We demonstrate the advantages of TouchCut by quantitatively comparing against the state of the art both in terms of accuracy, and run-time performance.

Storyboard sketches for Content Based Video Retrieval

We present a novel Content Based Video Retrieval (CBVR) system, driven by free-hand sketch queries depicting both objects and their movement (via dynamic cues; streak-lines and arrows). Our main contribution is a probabilistic model of video clips (based on Linear Dynamical Systems), leading to an algorithm for matching descriptions of sketched objects to video. We demonstrate our model fitting to clips under static and moving camera conditions, exhibiting linear and oscillatory motion. We evaluate retrieval on two real video data sets, and on a video data set exhibiting controlled variation in shape, color, motion and clutter.

Image and Video-based Artistic Stylisation

Non-photorealistic rendering (NPR) is a combination of computer graphics and computer vision that produces renderings in various artistic, expressive or stylized ways such as painting and drawing. This book focuses on image and video based NPR, where the input is a 2D photograph or a video rather than a 3D model. 2D NPR techniques have application in areas as diverse as consumer and professional digital photography and visual effects for TV and film production. The book covers the full range of the state of the art of NPR with every chapter authored by internationally renowned experts in the field, covering both classical and contemporary techniques. It will enable both graduate students in computer graphics, computer vision or image processing and professional developers alike to quickly become familiar with contemporary techniques, enabling them to apply 2D NPR algorithms in their own projects.

Probabilistic Motion Diffusion of Labeling Priors for Coherent Video Segmentation

We present a robust algorithm for temporally coherent video segmentation. Our approach is driven by multi-label graph cut applied to successive frames, fusing information from the current frame with an appearance model and labeling priors propagated forwarded from past frames. We propagate using a novel motion diffusion model, producing a per-pixel motion distribution that mitigates against cumulative estimation errors inherent in systems adopting “hard” decisions on pixel motion at each frame. Further, we encourage spatial coherence by imposing label consistency constraints within image regions (super-pixels) obtained via a bank of unsupervised frame segmentations, such as mean-shift. We demonstrate quantitative improvements in accuracy over state-of-the-art methods on a variety of sequences exhibiting clutter and agile motion, adopting the Berkeley methodology for our comparative evaluation.

The Relationship Between Systemising and Mental Rotation and the Implications for the Extreme Male Brain Theory of Autism

Within the Extreme Male Brain theory, Autism Spectrum Disorder is characterised as a deficit in empathising in conjunction with preserved or enhanced systemising. A male advantage in systemising is argued to underpin the traditional male advantage in mental rotation tasks. Mental rotation tasks can be separated into rotational and non-rotational components, and circulating testosterone has been found to consistently relate to the latter component. Systemising was found to correlate with mental rotation, specifically the non-rotational component(s) of the mental rotation task but not the rotational component of the task. Systemising also correlated with a proxy for circulating testosterone but not a proxy for prenatal testosterone. A sex difference was identified in systemising and the non-rotational aspect of the mental rotation task.

4D video textures for interactive character appearance

4D Video Textures (4DVT) introduce a novel representation for rendering video‐realistic interactive character animation from a database of 4D actor performance captured in a multiple camera studio. 4D performance capture reconstructs dynamic shape and appearance over time but is limited to free‐viewpoint video replay of the same motion. Interactive animation from 4D performance capture has so far been limited to surface shape only. 4DVT is the final piece in the puzzle enabling video‐realistic interactive animation through two contributions: a layered view‐dependent texture map representation which supports efficient storage, transmission and rendering from multiple view video capture; and a rendering approach that combines multiple 4DVT sequences in a parametric motion space, maintaining video quality rendering of dynamic surface appearance whilst allowing high‐level interactive control of character motion and viewpoint. 4DVT is demonstrated for multiple characters and evaluated both quantitatively and through a user‐study which confirms that the visual quality of captured video is maintained. The 4DVT representation achieves >90% reduction in size and halves the rendering cost.

A bag-of-regions approach to sketch-based image retrieval

This paper presents a system for retrieving photographs using free-hand sketched queries. Regions are extracted from each image by gathering nodes of a hierarchical image segmentation into a bag-of-regions (BoR) representation. The BoR represents object shape at multiple scales, encoding shape even in the presence of adjacent clutter. We extract a shape representation from each region, using the Gradient Field HoG (GF-HOG) descriptor which enables direct comparison with the sketched query. The retrieval pipeline yields significant performance improvements over the previous GF-HOG results reliant on single-scale Canny edge maps, and over leading descriptors (SIFT, SSIM) for visual search. In addition, our system enables localization of the sketched object within matching images.