Tobias Isenberg

Neural style transfer: a paradigm shift for image-based artistic rendering?

In this meta paper we discuss image-based artistic rendering (IB-AR) based on neural style transfer (NST) and argue, while NST may represent a paradigm shift for IB-AR, that it also has to evolve as an interactive tool that considers the design aspects and mechanisms of artwork production. IB-AR received significant attention in the past decades for visual communication, covering a plethora of techniques to mimic the appeal of artistic media. Example-based rendering represents one the most promising paradigms in IB-AR to (semi-)automatically simulate artistic media with high fidelity, but so far has been limited because it relies on pre-defined image pairs for training or informs only low-level image features for texture transfers. Advancements in deep learning showed to alleviate these limitations by matching content and style statistics via activations of neural network layers, thus making a generalized style transfer practicable. We categorize style transfers within the taxonomy of IB-AR, then propose a semiotic structure to derive a technical research agenda for NSTs with respect to the grand challenges of NPAR. We finally discuss the potentials of NSTs, thereby identifying applications such as casual creativity and art production.

Pondering the Concept of Abstraction in (Illustrative) Visualization

We explore the concept of abstraction as it is used in visualization, with the ultimate goal of understanding and formally defining it. Researchers so far have used the concept of abstraction largely by intuition without a precise meaning. This lack of specificity left questions on the characteristics of abstraction, its variants, its control, or its ultimate potential for visualization and, in particular, illustrative visualization mostly unanswered. In this paper we thus provide a first formalization of the abstraction concept and discuss how this formalization affects the application of abstraction in a variety of visualization scenarios. Based on this discussion, we derive a number of open questions still waiting to be answered, thus formulating a research agenda for the use of abstraction for the visual representation and exploration of data. This paper, therefore, is intended to provide a contribution to the discussion of the theoretical foundations of our field, rather than attempting to provide a completed and final theory.

Multiscale Visualization and Scale-Adaptive Modification of DNA Nanostructures

We present an approach to represent DNA nanostructures in varying forms of semantic abstraction, describe ways to smoothly transition between them, and thus create a continuous multiscale visualization and interaction space for applications in DNA nanotechnology. This new way of observing, interacting with, and creating DNA nanostructures enables domain experts to approach their work in any of the semantic abstraction levels, supporting both low-level manipulations and high-level visualization and modifications. Our approach allows them to deal with the increasingly complex DNA objects that they are designing, to improve their features, and to add novel functions in a way that no existing single-scale approach offers today. For this purpose we collaborated with DNA nanotechnology experts to design a set of ten semantic scales. These scales take the DNAs chemical and structural behavior into account and depict it from atoms to the targeted architecture with increasing levels of abstraction. To create coherence between the discrete scales, we seamlessly transition between them in a well-defined manner. We use special encodings to allow experts to estimate the nanoscale objects stability. We also add scale-adaptive interactions that facilitate the intuitive modification of complex structures at multiple scales. We demonstrate the applicability of our approach on an experimental use case. Moreover, feedback from our collaborating domain experts confirmed an increased time efficiency and certainty for analysis and modification tasks on complex DNA structures. Our method thus offers exciting new opportunities with promising applications in medicine and biotechnology.

DimSUM: Dimension and Scale Unifying Map for Visual Abstraction of DNA Origami Structures

We present a novel visualization concept for DNA origami structures that integrates a multitude of representations into a Dimension and Scale Unifying Map (DimSUM). This novel abstraction map provides means to analyze, smoothly transition between, and interact with many visual representations of the DNA origami structures in an effective way that was not possible before. DNA origami structures are nanoscale objects, which are challenging to model in silico. In our holistic approach we seamlessly combine three‐dimensional realistic shape models, two‐dimensional diagrammatic representations, and ordered alignments in one‐dimensional arrangements, with semantic transitions across many scales. To navigate through this large, two‐dimensional abstraction map we highlight locations that users frequently visit for certain tasks and datasets. Particularly interesting viewpoints can be explicitly saved to optimize the workflow. We have developed DimSUM together with domain scientists specialized in DNA nanotechnology. In the paper we discuss our design decisions for both the visualization and the interaction techniques. We demonstrate two practical use cases in which our approach increases the specialists’ understanding and improves their effectiveness in the analysis. Finally, we discuss the implications of our concept for the use of controlled abstraction in visualization in general.

Advances in visual computing: 12th international symposium, ISVC 2016 Las Vegas, NV, USA, december 12–14, 2016 proceedings, part II

Obesity is becoming a widely concerned health problem of most part of the world. Computer vision based recognition system has great potential to be an efficient tool to monitor food intake and cope with the growing problem of obesity. This paper proposes a food recognition algorithm based on sparse representation. The proposed algorithm learns overcomplete dictionaries from local descriptors including texture and color features that are extracted from food image patches. With the learned two overcomplete dictionaries, a feature vector of the food image can be generated with the sparsely encoded local descriptors. SVM is used for the classification. This research creates a Chinese food image dataset for experiments. Classifying Chinese food is more challenging because they are not as distinguishable visually as western food. The proposed algorithm achieves an average classification accuracy of 97.91% in a dataset of 5309 images that comprises 18 classes. The proposed method can be easily employed to dataset with more classes. Our results demonstrate the feasibility of using the proposed algorithm for food recognition.

Reducing affective responses to surgical images through color manipulation and stylization

We present the first empirical study on using color manipulation and stylization to make surgery images more palatable. While aversion to such images is natural, it limits many peoples ability to satisfy their curiosity, educate themselves, and make informed decisions. We selected a diverse set of image processing techniques, and tested them both on surgeons and lay people. While many artistic methods were found unusable by surgeons, edge-preserving image smoothing gave good results both in terms of preserving information (as judged by surgeons) and reducing repulsiveness (as judged by lay people). Color manipulation turned out to be not as effective.