Eun-Ji Lim

Design of distributed memory integration framework(DMIf)

Big memory applications such as in-memory database, denovo assembly application in the human genome sequencing area, big data analytics, and large scale scientific calculation are increasing explosively. However, the big memory system has been too expensive for many researchers and students. Therefore, methods to harvest remotely distributed memory has been considered as a cost effective way to run big memory applications in the cluster environment where computing nodes are connected via high speed network. We designed and implemented a new framework, DMIf(Distributed Memory Integrated framework), which harvests idle memory of distributed nodes in a cluster and serves it to the big memory application. The collected distributed memory will provide faster data processing in a cost-effective way for the big-data/memory application.

Design and implementation of user-level remote memory extension library

The increase of memory capacity has not kept up with the continuous increase of large memory applications. Therefore, approaches to utilize remote memory like a local memory has been considered as a cost effective way to run large memory application in the cluster environment where computing nodes are connected via high speed network. For the users of HPC cluster to run large memory application without administrators support, we suggest a user-level remote memory extension method. We designed and implemented a remote memory library model which extends the virtual address space of the large memory application process to remote memory. It includes user-level API and page fault handling mechanism, temporal page pool management and remote page prefetching algorithm. We also developed a performance test program to show if the user-level remote memory extension library works well. From the experimental test, we found that user-level remote memory extension library works well for applications with sequential access pattern.

A Design of Kernel-Level Remote Memory Extension System

Big data scientists are struggling with a memory capacity of the computer system because of the expansion speed of memory capacity has not kept up with the increasing requirement of large memory applications. Even though large memory application vitally requires big memory system, big memory machine has been too expensive for many researchers and students. By the way, as very high-speed networking technologies such as Infiniband EDR(100 Gbps) have been developed, approaches to utilize remote memory has been considered as a cost effective way to run large memory applications in the HPC cluster environment. For the general users of HPC cluster system who want to run large memory application with administrator’s support, we suggest a kernel-level remote memory extension system. We designed a remote memory extension system which mapped remote memory pages to the virtual address space of the large memory application process. The system includes three components such as remote memory consumer, Integrated Memory Manager, and memory provider. We developed a kernel-level remote memory extension device and achieved 4 × improvement of the latency of page fault handling on remote memory.

Design of Cache Backend Using Remote Memory for Network File System

In supporting high-performance data processing, performance gap between the computation device and storage prevents the full utilization of the computation resource and causes a system bottleneck. In addition, some big-data applications which require interactive, real-time, and complicated computation need faster data I/O than distributed file systems. So we propose a new cache backend facility called CacheDM for network file system, which utilizes the distributed memory as a cache media in the cluster environment where computing nodes are connected via high speed network. CacheDM can provide low-latency and high-speed cache by supporting direct memory copy based access to the cached data by using RDMA. CacheDM is designed as a cache backend of the FS-Cache, therefore users can use CacheDM and gain its performance advantage without modification of existing application and NFS.

A study on user-level remote memory extension system

The speed of memory capacity expansion of the computer system has not kept up with the speed of the increase of the memory requirement of large memory applications. Also, big memory system has been too expensive for many researchers and students. Therefore, approaches to utilize remote memory has been considered as a cost effective way to run large memory applications in the cluster environment where computing nodes are connected via high speed network. For the general users of HPC cluster system who want to run large memory application without administrators support, we suggest a user-level remote memory extension method. We designed and implemented a remote memory extension system which mapped remote memory pages to the virtual address space of the large memory application process. The system includes three components such as remote memory consumer, integrated memory manager, and memory provider. We also developed a test program and did experiment to show how good performance the user-level remote memory extension system can achieves. Through the experimental test, we will show the measured latency and bandwidth of user-level remote memory extension system for sequential access pattern and random access pattern.

A software integration architecture for support of the large-capacity memory

This article is concerned chiefly with a software integration architecture for providing a large-capacity collective memory layer on a multi-node system by using a remote direct memory access technique and a software virtualization technique. Major components described by the architecture include a memory region management module, a large-capacity collective memory service module, and a data cooperative module. As the effect of the architecture, the following merits are considered: (1) shorten access time more than a disk-based storage, (2) input /output delay prevention by prefetching mechanism, (3) increased system scalability and (4) burst real-time stream computation.

Self-provisioning and configuration system for heterogeneous diskless cluster system

As a platform for cloud services and high performance computing, cluster systems are primarily used. As the scale of the cluster system is increasing more and more and the use of heterogeneous nodes equipped with various hardware is increased, it is becoming more difficult and exhausting in time and cost to manage a cluster system in a conventional manner. In this paper, we design self-provisioning and configuration system to configure, setup, and perform provisioning for cluster system consisting of heterogeneous diskless nodes. We expect our system to make cluster system management easy and reduce time and costs to manage a cluster system.

A design of resource fault handling mechanism using dynamic resource reallocation for the resource and job management system

Due to the development of NGS technologies and the reduction of analysis cost, it is possible to perform population-scale human genome analysis. Also, large amount of genome data have been exploded recently. It is required for introduction parallel processing using High Performance Computing systems to analyse and handle these large data through genome analysis pipeline. In this paper, we propose the resource fault handling mechanism based on dynamic resource reconfiguration and delayed scheduling for data-intensive pipeline job processing such as genome analysis executed on the large cluster systems interconnected by high speed and low latency network. In order to prevent the abnormal job completion caused by lack of the specific resources, we offer the resource fault detection and handling methods. If the cause of fault is lack of resources, it can be solved by the resource re-allocation and process freezing/resuming based delayed job execution or process migration on the available node.

Design and implementation of forced unmount

This paper describes a kernel function named FU (forced unmount) related to file system on Linux system. FU is a function to forcibly unmount file systems in despite of busy state of file systems. The current implementation was developed on Linux-2.6.8 and tested by tools, POSTMAR and LTP. This contains considerations of FU and a algorithm to solve problems during developing FU.