Bret Jackson

Lift-Off: Using Reference Imagery and Freehand Sketching to Create 3D Models in VR

Three-dimensional modeling has long been regarded as an ideal application for virtual reality (VR), but current VR-based 3D modeling tools suffer from two problems that limit creativity and applicability: (1) the lack of control for freehand modeling, and (2) the difficulty of starting from scratch. To address these challenges, we present Lift-Off, an immersive 3D interface for creating complex models with a controlled, handcrafted style. Artists start outside of VR with 2D sketches, which are then imported and positioned in VR. Then, using a VR interface built on top of image processing algorithms, 2D curves within the sketches are selected interactively and “lifted” into space to create a 3D scaffolding for the model. Finally, artists sweep surfaces along these curves to create 3D models. Evaluations are presented for both long-term users and for novices who each created a 3D sailboat model from the same starting sketch. Qualitative results are positive, with the visual style of the resulting models of animals and other organic subjects as well as architectural models matching what is possible with traditional fine art media. In addition, quantitative data from logging features built into the software are used to characterize typical tool use and suggest areas for further refinement of the interface.

A Lightweight Tangible 3D Interface for Interactive Visualization of Thin Fiber Structures

We present a prop-based, tangible interface for 3D interactive visualization of thin fiber structures. These data are commonly found in current bioimaging datasets, for example second-harmonic generation microscopy of collagen fibers in tissue. Our approach uses commodity visualization technologies such as a depth sensing camera and low-cost 3D display. Unlike most current uses of these emerging technologies in the games and graphics communities, we employ the depth sensing camera to create a fish-tank sterePoscopic virtual reality system at the scientists desk that supports tracking of small-scale gestures with objects already found in the work space. We apply the new interface to the problem of interactive exploratory visualization of three-dimensional thin fiber data. A critical task for the visual analysis of these data is understanding patterns in fiber orientation throughout a volume.The interface enables a new, fluid style of data exploration and fiber orientation analysis by using props to provide needed passive-haptic feedback, making 3D interactions with these fiber structures more controlled. We also contribute a low-level algorithm for extracting fiber centerlines from volumetric imaging. The system was designed and evaluated with two biophotonic experts who currently use it in their lab. As compared to typical practice within their field, the new visualization system provides a more effective way to examine and understand the 3D bioimaging datasets they collect.

Toward mixed method evaluations of scientific visualizations and design process as an evaluation tool

In this position paper we discuss successes and limitations of current evaluation strategies for scientific visualizations and argue for embracing a mixed methods strategy of evaluation. The most novel contribution of the approach that we advocate is a new emphasis on employing design processes as practiced in related fields (e.g., graphic design, illustration, architecture) as a formalized mode of evaluation for data visualizations. To motivate this position we describe a series of recent evaluations of scientific visualization interfaces and computer graphics strategies conducted within our research group. Complementing these more traditional evaluations our visualization research group also regularly employs sketching, critique, and other design methods that have been formalized over years of practice in design fields. Our experience has convinced us that these activities are invaluable, often providing much more detailed evaluative feedback about our visualization systems than that obtained via more traditional user studies and the like. We believe that if design-based evaluation methodologies (e.g., ideation, sketching, critique) can be taught and embraced within the visualization community then these may become one of the most effective future strategies for both formative and summative evaluations.

Supporting internal visualization of biomedical datasets via 3D rapid prototypes and sketch-based gestures

In this work, we explore the potential of combining virtual reality (VR) visualizations of data with physical models generated via rapid 3D prototyping to provide a new style of exploratory data visualization. Holding a physical rapid prototype model in ones hand can provide an immediate and more accurate understanding of a complex 3D form than can be provided via a computer display, even a head-tracked VR display. However, physical printouts are static and only show the bounding surface (the outside) of a 3D geometry. When working with many of todays scientific datasets, for example, analyzing results of high-performance simulations of cardiovascular fluid dynamics [Simon et al. 2010], scientists need to understand both a complex bounding surface for the data and multivariate volumetric data contained within it. We present the initial design of a new interface for exploring this type of data. Our goal is to combine the intuitive shape understanding made possible by physical 3D rapid prototypes with complementary VR visualizations of the data inside the printed geometry.

Force Brushes: Progressive Data-Driven Haptic Selection and Filtering for Multi-Variate Flow Visualizations

We present Force Brushes, a haptic-based interaction technique for controlled selection in multi-variate flow visualizations. Force Brushes addresses the difficult task of volumetric selection and filtering by rendering haptic constraints that allow scientists to snap directly to proxy geometry, such as streamlines, to select regions of interest and then progressively filter the selection using a data-driven approach. Using progressive brushing actions with multiple variables, a user has the potential to explore volumetric data in a more immediate, fluid, and controllable way guided by the underlying data.

Cartograph: Unlocking Spatial Visualization Through Semantic Enhancement

This paper introduces Cartograph, a visualization system that harnesses the vast amount of world knowledge encoded within Wikipedia to create thematic maps of almost any data. Cartograph extends previous systems that visualize non-spatial data using geographic approaches. While these systems required data with an existing semantic structure, Cartograph unlocks spatial visualization for a much larger variety of datasets by enhancing input datasets with semantic information extracted from Wikipedia. Cartographs map embeddings use neural networks trained on Wikipedia article content and user navigation behavior. Using these embeddings, the system can reveal connections between points that are unrelated in the original data sets, but are related in meaning and therefore embedded close together on the map. We describe the design of the system and key challenges we encountered, and we present findings from an exploratory user study

Immersive Analytics for Medicine: Hybrid 2D/3D Sketch-Based Interfaces for Annotating Medical Data and Designing Medical Devices

We explore the role that immersive technologies, specifically virtual reality (VR) and hybrid 2D/3D sketch-based interfaces and visualizations, can play in analytical reasoning for medicine. Two case studies are described: (1) immersive explanations of medical procedures, and (2) immersive design of medical devices. Both tightly integrate 2D imagery and data with 3D interfaces, models, and visualizations. This is an approach we argue is likely to be particularly useful in medicine, where analytical tasks often involve relating 2D data (e.g., medical imaging) to 3D contexts (e.g., a patients body). User feedback and observations from our interdisciplinary team indicate the utility of the approach for the current case studies as well as some shortcomings and areas for future research. This work contributes to a broader discussion of how hybrid 2D/3D interfaces may form an essential ingredient of future immersive analytics systems across a variety of domains.