Jessica Trelogan

A case study on entity Resolution for Distant Processing of big Humanities data

At the forefront of big data in the Humanities, collections management can directly impact collections access and reuse. However, curators using traditional data management methods for tasks such as identifying redundant from relevant and related records, a small increase in data volume can significantly increase their workload. In this paper, we present preliminary work aimed at assisting curators in making important data management decisions for organizing and improving the overall quality of large unstructured Humanities data collections. Using Entity Resolution as a conceptual framework, we created a similarity model that compares directories and files based on their implicit metadata, and clusters pairs of closely related directories. Useful relationships between data are identified and presented through a graphical user interface that allows qualitative evaluation of the clusters and provides a guide to decide on data management actions. To evaluate the models performance, we experimented with a test collection and asked the curator to classify the clusters according to four model cluster configurations that consider the presence of related and duplicate information. Evaluation results suggest that the model is useful for making data management action decisions.

Using High Performance Computing for Detecting Duplicate, Similar and Related Images in a Large Data Collection

The detection of duplicate and related content is a critical data curation task in the context of digital research collections. This task can be challenging, if not impossible, to do manually in large, unstructured, and noisy collections. While there are many automated solutions for deduplicating data that contain large numbers of identical copies, it can be particularly difficult to find a solution for identifying redundancy within image-heavy collections that have evolved over a long span of time or have been created collaboratively by large groups. These types of collections, especially in academic research settings, in which the datasets are used for a wide range of publication, teaching, and research activities, can be characterized by (1) large numbers of heterogeneous file formats, (2) repetitive photographic documentation of the same subjects in a variety of conditions (3) multiple copies or subsets of images with slight modifications (e.g., cropping or color-balancing) and (4) complex file structures and naming conventions that may not be consistent throughout. In this chapter, we present a scalable and automated approach for detecting duplicate, similar, and related images, along with subimages, in digital data collections. Our approach can assist in efficiently managing redundancy in any large image collection on High Performance Computing (HPC) resources. While we illustrate the approach with a large archaeological collection, it is domain-neutral and is widely applicable to image-heavy collections within any HPC platform that has general-purpose processors.

Cyberinfrastructure for Digital Libraries and Archives: Integrating Data Management, Analysis, and Publication

Increasingly, digital libraries and archives need to and are using cyberinfrastructure and machine learning to meet curation, data management, and researchers needs. This workshop focuses on facilitating adoption and integration between these spaces. It brings together researchers and practitioners to share visions, questions, latest advances in methodology, application experiences, and best practices.

Connecting the non-traditional user-community to the national CyberInfrastructure

This paper reports on a hands-on workshop that was organized to promote the usage of the national CyberInfrastructure (CI) amongst non-traditional High Performance Computing (HPC) users. The majority of the workshop participants were students and professionals who had never used the CI before but were interested in leveraging it for conducting computationally-intensive Big Data management activities. With the support from the National Science Foundation (NSF), students from underrepresented groups were also funded to participate in the workshop, where they learnt about both the CI and Big Data management. The workshop itself was an outcome of an XSEDE Extended Collaborative Support Service (ECSS) project that involved non-traditional HPC users from the archaeology domain.