Sean Kandel

Research directions in data wrangling: visuatizations and transformations for usable and credible data

In spite of advances in technologies for working with data, analysts still spend an inordinate amount of time diagnosing data quality issues and manipulating data into a usable form. This process of ‘data wrangling’ often constitutes the most tedious and time-consuming aspect of analysis. Though data cleaning and integration arelongstanding issues in the database community, relatively little research has explored how interactive visualization can advance the state of the art. In this article, we review the challenges and opportunities associated with addressing data quality issues. We argue that analysts might more effectively wrangle data through new interactive systems that integrate data verification, transformation, and visualization. We identify a number of outstanding research questions, including how appropriate visual encodings can facilitate apprehension of missing data, discrepant values, and uncertainty; how interactive visualizations might facilitate data transform specification; and how recorded provenance and social interaction might enable wider reuse, verification, and modification of data transformations.

Profiler: integrated statistical analysis and visualization for data quality assessment

Data quality issues such as missing, erroneous, extreme and duplicate values undermine analysis and are time-consuming to find and fix. Automated methods can help identify anomalies, but determining what constitutes an error is context-dependent and so requires human judgment. While visualization tools can facilitate this process, analysts must often manually construct the necessary views, requiring significant expertise. We present Profiler, a visual analysis tool for assessing quality issues in tabular data. Profiler applies data mining methods to automatically flag problematic data and suggests coordinated summary visualizations for assessing the data in context. The system contributes novel methods for integrated statistical and visual analysis, automatic view suggestion, and scalable visual summaries that support real-time interaction with millions of data points. We present Profilers architecture --- including modular components for custom data types, anomaly detection routines and summary visualizations --- and describe its application to motion picture, natural disaster and water quality data sets.

Interactive analysis of big data

New user interfaces can transform how we work with big data, and raise exciting research problems that span human-computer interaction, machine learning, and distributed systems.

Data collection by the people, for the people

Data Collection by the People, for the People is a CHI 2011 workshop to explore data from the crowd, bringing together mobile crowdsourcing & participatory urbanism researchers with data analysis and visualization researchers. The workshop is two-day event beginning with day of field work in the city of Vancouver, trying out mobile crowdsourcing applications and data analysis tools. Participants are encouraged to contribute applications and tools which they wish to share. Our goal is to provoke discussion and brainstorming, enabling both data collection researchers and data manipulationanalysis researchers to benefit from mutually learned lessons about crowdsourced data.