Generating insights through data preparation, visualization, and analysis: Framework for combining clustering and data visualization techniques for low-cardinality sequential data

Abstract

Abstract In this paper, we introduce a novel approach for identifying and testing relationships and patterns on the types of sequential data that are broadly present in a number of different real-world scenarios and environments. The proposed two-phase framework combines data preparation, data visualization and clustering techniques in an innovative way. The first phase of the framework explores the large amount of sequential data in stages that can be undertaken iteratively. Those stages include data preparation, counting and value-based ordering, distribution visualization, and subsequence length determination, confirmation and re-visualization. The second phase of the framework explores sequence differences, based on motifs, between data cohorts that are created using descriptive attributes, and visualizes the changes over time and different attribute values. To illustrate the analytical power of the proposed framework, we present a comprehensive example that applies the framework on a large formally-maintained research data set collected and managed by the US Census Bureau. The framework, and the presented example, utilize visualization as an analytics tool and not just a presentation accessory.