Thomas Renner

People-Centric Internet of Things—Challenges, Approach, and Enabling Technologies

Technology now offers the possibility of delivering a vast range of low-cost people-centric services to citizens. Internet of Things (IoT) supporting technologies are becoming robust, viable and cheaper. Mobile phones are increasingly more powerful and disseminated. On the other hand, social networks and virtual worlds are experiencing an exploding popularity and have millions of users. These low-cost technologies can now be used to create an Internet of People (IoP), a dynamically configurable integration platform of connected smart objects that allows enhanced, people-centric applications. As opposed to things-centric ones, IoP combines the real, sensory world with the virtual world for the benefit of people while it also enables the development of sensing applications in contexts such as e-health, sustainable mobility, social networks enhancement or fulfilling people’s special needs. This paper identifies the main challenges, a possible approach, and key enabling technologies for a people-centric society based on the Internet of Things.

The Device Cloud - Applying Cloud Computing Concepts to the Internet of Things

The pervasiveness of connected embedded devices and Internet of Things (IoT) related application domains like smart cities, e-Health or, transportation lead to an constantly increasing amount of data, compute- and storage resources surrounding us. However, currently there is a gap between data acquisition and processing, usually bridged by gateway based approaches that integrate the devices and forward the data to Clouds in order to be processed. Integration is often static and limited to a certain application domain, which the gateway is able to support. In addition, processing does not consider the compute- and storage resources already made available by the growing amount of smart devices, like smart TVs or smart phones. As a solution, we propose the Device Cloud, which can be envisioned as an application of the Cloud Computing Infrastructure as a Service (IaaS) and Platform as a Service (PaaS) paradigms to the IoT domain. The approach will allow sharing and on demand provisioning of resources provided by the connected embedded devices surrounding us. Besides introducing an architecture draft to provide a common understanding of the overall problem, the main focus of the paper is to discuss challenges that arise and give a state of the art review of related research initiatives that are eligible to contribute to possible solutions.

Towards Container-Based Resource Management for the Internet of Things

The Internet of Things (IoT) paradigm gains momentum for vendors, developers and users. A variety of available devices and technologies promote the deployment of solutions and applications in various domains. However, the increasing amount of IoT devices leads to an increasing amount of resources made available to the users. If devices like smart phones or smart TVs are considered, this includes computing and storage resources. In order to increase the utilization of these IoT resources and reduce the amount of generated network traffic, we propose a container-based resource allocation scheme. The approach allows various applications and users to dynamically allocate resources offered by edge devices and process IoT data close to the source. The approach is evaluated regarding its feasibility in terms of performance on resource constrained IoT devices.

Portable SDN Applications on the PaaS Layer

SaaS applications often face a vendor or technical lock-in due to PaaS provider specific specifications, like cloud management APIs. As a solution, this paper presents a novel approach for developing applications more PaaS provider independent. In particular, the approach illustrates advantages of extending JavaEE application servers with a new container that executes so called Infrastructure Java Beans and provides a new set of state-of-the-art cloud APIs, which for instance enable Software Defined Networking capabilities. This container allows to manage and observe the application server itself as well as the cloud infrastructure, including compute and network resources. Thus, the application server can scale on demand controlled by the application itself and adjusted with optimized network functions that are designed application specific. In addition, all of this advantages are invested with a cloud provider independent migration possibility.

Towards On-Demand Resource Provisioning for IoT Environments

The set of connected embedded devices surrounding and providing resources to us will constantly grow in the future. Currently these devices are often treated as pure data sources and there is no notion of providing and provisioning the compute and storage resources they offer to the users. Paradigms like Infrastructure as a Service, Platform as a Service or, Pay as you Go are very popular and successful in the Cloud Computing domain. We will discuss whether these paradigms can be applied to the Internet of Things domain in order to create a Cloud of Things that is surrounding us and provides resources in an ubiquitous fashion.

Visually programming dataflows for distributed data analytics

Distributed dataflow systems like Spark and Flink allow to analyze large datasets using clusters of computers. These frameworks provide automatic program parallelization and manage distributed workers, including worker failures. Moreover, they provide high-level programming abstractions and execute programs efficiently. Yet, the programming abstractions remain textual while the dataflow model is essentially a graph of transformations. Thus, there is a mismatch between the presented abstraction and the underlying model here. One can also argue that developing dataflow programs with these textual abstractions requires needless amounts of coding and coding skills. A dedicated programming environment could instead allow constructing dataflow programs more interactively and visually. In this paper, we therefore investigate how visual programming can make the development of parallel dataflow programs more accessible. In particular, we built a prototypical visual programming environment for Flink, which we call Flision. Flision provides a graphical user interface for creating dataflow programs, a code generation engine that generates code for Flink, and seamless deployment to a connected cluster. Users of this environment can effectively create jobs by dragging, dropping, and visually connecting operator components. To evaluate the applicability of this approach, we interviewed ten potential users. Our impressions from this qualitative user testing strengthened our believe that visual programming can be a valuable tool for users of scalable data analysis tools.

Network-aware resource management for scalable data analytics frameworks

Sharing cluster resources between multiple frameworks, applications and datasets is important for organizations doing large scale data analytics. It improves cluster utilization, avoids standalone clusters running only a single framework and allows data scientists to choose the best framework for each analysis task. Current systems for cluster resource management like YARN or Mesos achieve resource sharing using containers. Analytics frameworks execute their tasks in these containers. However, currently the container placement is based predominantly on available computing capabilities in terms of cores and memory, yet neglects to also take the network topology and data locations into account. In this paper, we propose a container placement approach that (a) takes the network topology into account to prevent network congestions in the core network and (b) places containers close to input data to improve data locality and reduce remote disk reads in distributed file systems. The main advantages of introducing topology- and data-awareness on the level of container placement is that multiple application frameworks benefit from improvements. We present a prototype integrated with Hadoop YARN and an evaluation with workloads consisting of different applications and datasets using Apache Flink. Our evaluation on a 64 core cluster, in which nodes are connected through a fat tree topology, shows promising results with speedups of up to 67% for network-intensive workloads.

Scheduling Recurring Distributed Dataflow Jobs Based on Resource Utilization and Interference

Resource management systems like YARN or Mesos enable users to share cluster infrastructures by running analytics jobs in temporarily reserved containers. These containers are typically not isolated to achieve high degrees of overall resource utilizations despite the often fluctuating resource usage of single analytic jobs. However, some combinations of jobs utilize the resources better and interfere less with each others when running on the same nodes than others. This paper presents an approach for improving the resource utilization and job throughput when scheduling recurring data analysis jobs in shared cluster environments. Using a reinforcement learning algorithm, the scheduler continuously learns which jobs are best executed simultaneously on the cluster. Our evaluation of an implementation built on Hadoop YARN shows that this approach can increase resource utilization and decrease job runtimes. While interference between jobs can be avoided, co-locations of jobs with complementary resource usage are not yet always fully recognized. However, with a better measure of co-location goodness, our solution can be used to automatically adapt the scheduling to workloads with recurring batch jobs.

Adaptive Resource Management for Distributed Data Analytics based on Container-level Cluster Monitoring.

Many distributed data analysis jobs are executed repeatedly in production clusters. Examples include daily executed batch jobs and iterative programs. These jobs present an opportunity to learn workload characteristics through continuous fine-grained cluster monitoring. Therefore, based on detailed profiles of resource utilization, data placement, and job runtimes, resource management can in fact adapt to actual workloads. In this paper, we present a system architecture that contains four mechanisms for an adaptive resource management, encompassing data placement, resource allocation, and container as well as job scheduling. In particular, we extended Apache Hadoop’s scheduling and data placement to improve resource utilization and job runtimes for recurring analytics jobs. Furthermore, we developed a Hadoop submission tool that allows users to reserve resources for specific target runtimes and which uses historical data available from cluster monitoring

Ellis: Dynamically Scaling Distributed Dataflows to Meet Runtime Targets

Distributed dataflow systems like MapReduce, Spark, and Flink help users in analyzing large datasets with a set of cluster resources. Performance modeling and runtime prediction is then used for automatically allocating resources for specific performance goals. However, the actual performance of distributed dataflow jobs can vary significantly due to factors like interference with co-located workloads, varying degrees of data locality, and failures.We address this problem with Ellis, a system that allocates an initial set of resources for a specific runtime target, yet also continuously monitors a jobs progress towards the target and if necessary dynamically adjusts the allocation. For this, Ellis models the scale-out behavior of individual stages of distributed dataflow jobs based on previous executions. Our evaluation of Ellis with iterative Spark jobs shows that dynamic adjustments can reduce the number of constraint violations by 30.7-75.0% and the magnitude of constraint violations by 70.6-94.5%.