Joshua Eckroth

Teaching Big Data with a Virtual Cluster

Both industry and academia are confronting the challenge of big data, i.e., data processing that involves data so voluminous or arriving at such high velocity that no single commodity machine is capable of storing or processing them all. A common approach to handling big data is to divide and distribute the processing job to a cluster of machines. Ideally, a course that teaches students how to work with big data would provide students access to a cluster for hands-on practice. However, a cluster of physical, on-premise machines may be prohibitively expensive, particularly at smaller institutions with smaller budgets. In this report, we summarize our experiences developing and using a virtual cluster in a big data mining and analytics course at a small private liberal arts college. A single moderately-sized server hosts a cluster of virtual machines, which run the popular Apache Hadoop system. The virtual cluster gives students hands-on experience and costs less than an equal number of physical machines. It is also easily constructed and reconfigured. We describe our implementation, analyze its performance characteristics, and compare costs with physical clusters and the Amazon Elastic MapReduce cloud service. We summarize our use of the virtual cluster in the classroom and show student feedback.

Improving software quality from the requirements specification

The first stage of software development, functional requirements specification, is considered the most important stage in the software lifecycle. Requirements constructed in this stage affect all other stages of the lifecycle, and thus affect software quality. We provide a method for determining how functional requirements affect software quality. To do so, we utilize a functional modeling framework that includes a controlled language for requirements specification and assess software qualities. Then we apply an information entropy metric to measure the significance of each software requirement. Using this method the designer can identify which requirements, when implemented, will most affect software quality.

Toward a science of design for software-intensive systems

This research is intended as an initial step toward the development of a rigorous science of design for software-intensive systems. Our work, based on existing research, has provided new definitions for design, science of design, and software-intensive systems. By identifying design issues currently affecting the field, we will show why software-intensive systems require a science of design. We have also provided some functional requirements for a science of design, what it must address, and how it might do so. We anticipate that these definitions and ideas will be embraced within the research community, and be utilized as a foundation for continued investigation. This work is a starting point for future research toward a science of design with extensive references and ideas for further research.

Building AI Applications: Yesterday, Today, and Tomorrow

AI applications have been deployed and used for industrial, government, and consumer purposes for many years. The experiences have been documented in IAAI conference proceedings since 1989. Over the years, the breadth of applications has expanded many times over and AI systems have become more commonplace. Indeed, AI has recently become a focal point in the industrial and consumer consciousness. This article focuses on changes in the world of computing over the last three decades that made building AI applications more feasible. We then examine lessons learned during this time and distill these lessons into succinct advice for future application builders.

NewsFinder: Automating an AI News Service

NewsFinder automates the steps involved in finding, selecting, categorizing, and publishing news stories that meet relevance criteria for the Artificial Intelligence community. The software combines a broad search of online news sources with topic-specific trained models and heuristics. Since August 2010, the program has been used to operate the AI in the News service that is part of the AAAI AITopics website.

A course on big data analytics

This report details a course on big data analytics designed for undergraduate junior and senior computer science students. The course is heavily focused on projects and writing code for big data processing. It is designed to help students learn parallel and distributed computing frameworks and techniques commonly used in industry. The curriculum includes a progression of projects requiring increasingly sophisticated big data processing ranging from data preprocessing with Linux tools, distributed processing with Hadoop MapReduce and Spark, and database queries with Hive and Googles BigQuery. We discuss hardware infrastructure and experimentally evaluate the cost/benefit of an on-premise server versus Amazons Elastic MapReduce. Finally, we showcase outcomes of our course in terms of student engagement and anonymous student feedback. A course designed for undergraduate junior and senior computer science students.Curriculum is project-oriented and focuses on modern tools such as Hadoop and Spark.Course projects are explained in detail.On-premise and cloud computing infrastructure are evaluated and cost-compared.

AI education: adaptive planning

In this column, we focus on designing assignments and projects that make use of planning engines. Planning has been one of the pillars of artificial intelligence since the origin of the field, and the research community remains active, as evidenced by competitions such as the IPC and conferences such as ICAPS.