Design and Implementation of a Universal Data Quality Management Software Based on Data Flow

Abstract

Data quality problems are analyzed to get several typical problems, such as data missing, data duplication, data abnormality, data inconsistency and data logic error. In order to resolve these problems, a universal data quality management software is proposed. This software provides data cleaning method for each problem and evaluates the effect of these methods, and manages the plug-in components for use. The architecture and critical technologies are introduced in detail, and main steps are shown with a specific application. According to the feedback of users, this new software is powerful, adaptable, simple to use, easy to set processes, with high data cleaning efficiency and accurate effect evaluations.