Emre Kiciman

Pinpoint: problem determination in large, dynamic Internet services

Traditional problem determination techniques rely on static dependency models that are difficult to generate accurately in todays large, distributed, and dynamic application environments such as e-commerce systems. We present a dynamic analysis methodology that automates problem determination in these environments by 1) coarse-grained tagging of numerous real client requests as they travel through the system and 2) using data mining techniques to correlate the believed failures and successes of these requests to determine which components are most likely to be at fault. To validate our methodology, we have implemented Pinpoint, a framework for root cause analysis on the J2EE platform that requires no knowledge of the application components. Pinpoint consists of three parts: a communications layer that traces client requests, a failure detector that uses traffic-sniffing and middleware instrumentation, and a data analysis engine. We evaluate Pinpoint by injecting faults into various application components and show that Pinpoint identifies the faulty components with high accuracy and produces few false-positives.

Detecting application-level failures in component-based Internet services

Most Internet services (e-commerce, search engines, etc.) suffer faults. Quickly detecting these faults can be the largest bottleneck in improving availability of the system. We present Pinpoint, a methodology for automating fault detection in Internet services by: 1) observing low-level internal structural behaviors of the service; 2) modeling the majority behavior of the system as correct; and 3) detecting anomalies in these behaviors as possible symptoms of failures. Without requiring any a priori application-specific information, Pinpoint correctly detected 89%-96% of major failures in our experiments, as compared with 20%-70% detected by current application-generic techniques.

ICEBERG: an Internet core network architecture for integrated communications

In the ICEBERG project at UC Berkeley, we are developing an Internet-based integration of telephony and data services spanning diverse access networks. Our primary goals are extensibility, scalability, robustness, and personalized communication. We leverage the Internets low cost of entry for service creation, provision, deployment, and integration. We present our solutions to signaling, easy service creation, resource reservation, admission control, billing, and security in the ICEBERG network architecture.

JAGR: an autonomous self-recovering application server

This paper demonstrates that the dependability of generic, evolving J2EE applications can be enhanced through a combination of a few recovery-oriented techniques. Our goal is to reduce downtime by automatically and efficiently recovering from a broad class of transient software failures without having to modify applications. We describe here the integration of three new techniques into JBoss, an open-source J2EE application server. The resulting system is JAGR-JBoss with application-generic recovery - a self-recovering execution platform. JAGR combines application-generic failure-path inference (AFPI), path-based failure detection, and micro-reboots. AFPI uses controlled fault injection and observation to infer paths that faults follow through a J2EE application. Path-based failure detection uses tagging of client requests and statistical analysis to identify anomalous component behavior. Micro-reboots are fast reboots we perform at the sub-application level to recover components from transient failures; by selectively rebooting only those components that are necessary to repair the failure, we reduce recovery time. These techniques are designed to be autonomous and application-generic, making them well suited to the rapidly changing software of Internet services.

Portability, extensibility and robustness in iROS

The dynamism and heterogeneity in ubicomp environments on both short and long time scales implies that middleware platforms for these environments need to be designed ground up for portability, extensibility and robustness. In this paper, we describe how we met these requirements in iROS, a middleware platform for a class of ubicomp environments, through the use of three guiding principles - economy of mechanism, client simplicity and levels of indirection. Apart from theoretical arguments and experimental results, experience through several deployments with a variety of apps, in most cases not done by the original designers of the system, provides some validation in practice that the design decisions have in fact resulted in the intended portability, extensibility and robustness. A retrospective examination of the system leads the authors to the following lesson: A logically-centralized design and physically-centralized implementation enables the best behavior in terms of extensibility and portability along with ease of administration, and sufficient behavior in terms of scalability and robustness.

The Roma personal metadata service

People now have available to them a diversity of digital storage facilities, including laptops, cell phone address books, handheld devices, desktop computers and web-based storage services. Unfortunately, as the number of personal data repositories increases, so does the management problem of ensuring that the most up-to-date version of any document in a users personal file space is available to him on the storage facility he is currently using. We introduce the Roma personal metadata service to make it easier to locate current versions of personal files and ensure their availability across different repositories. This centralized service stores information about each of a users files, such as name, location, timestamp and keywords, on behalf of mobility-aware applications. Separating out these metadata from the data respositories makes it practical to keep the metadata store on a highly available, portable device. In this paper we describe the design requirements, architecture and current prototype implementation of Roma.

Discovering Shifts to Suicidal Ideation from Mental Health Content in Social Media

History of mental illness is a major factor behind suicide risk and ideation. However research efforts toward characterizing and forecasting this risk is limited due to the paucity of information regarding suicide ideation, exacerbated by the stigma of mental illness. This paper fills gaps in the literature by developing a statistical methodology to infer which individuals could undergo transitions from mental health discourse to suicidal ideation. We utilize semi-anonymous support communities on Reddit as unobtrusive data sources to infer the likelihood of these shifts. We develop language and interactional measures for this purpose, as well as a propensity score matching based statistical approach. Our approach allows us to derive distinct markers of shifts to suicidal ideation. These markers can be modeled in a prediction framework to identify individuals likely to engage in suicidal ideation in the future. We discuss societal and ethical implications of this research.

Using Dynamic Mediation to Integrate COTS Entities in a Ubiquitous Computing Environment

The original vision of ubiquitous computing [14] is about enabling people to more easily accomplish tasks through the seamless interworking of the physical environment and a computing infrastructure. A major challenge to the practical realization of this vision involves the integration of commercial-off-the-shelf (COTS) hardware and software components: consider the awkwardness of such a mundane task as exporting a textual memo written on a Palm Pilot to a Microsoft Word document.It is not enough to overcome the protocol and data format mismatches that currently impede the interoperation of these entities: for the user experience to be truly seamless, we must provide a framework for the dynamic connection of such endpoints on demand, to support the ad-hoc interactions that are an integral part of ubiquitous computing. To this end, we offer a dynamic mediation framework called Paths. A Path consists of dynamically instantiated, automatically composable operators that bridge datatype and protocol mismatches between components wishing to communicate.Because operator composability is inferred from the type system, adding support for a new type of endpoint requires only incremental work; because the control and data flow for Paths are largely decoupled from the communicating endpoints, it is easy to connect COTS or legacy components. We describe the Paths architecture, our prototype implementation, and our experience and lessons based on several production applications built with the framework, and outline some continuing work on Paths in the context of the Stanford Interactive Workspaces project.

Autonomous recovery in componentized Internet applications

In this paper we show how to reduce downtime of J2EE applications by rapidly and automatically recovering from transient and intermittent software failures, without requiring application modifications. Our prototype combines three application-agnostic techniques: macroanalysis for fault detection and localization, microrebooting for rapid recovery, and external management of recovery actions. The individual techniques are autonomous and work across a wide range of componentized Internet applications, making them well-suited to the rapidly changing software of Internet services. The proposed framework has been integrated with JBoss, an open-source J2EE application server. Our prototype provides an execution platform that can automatically recover J2EE applications within seconds of the manifestation of a fault. Our system can provide a subset of a systems active end users with the illusion of continuous uptime, in spite of failures occurring behind the scenes, even when there is no functional redundancy in the system.

Discovering correctness constraints for self-management of system configuration

Managing the configuration of computer systems is a difficult task. Too easily, a computer user or administrator can make a simple mistake and misconfigure a system, causing instabilities, unexpected behavior, and general unreliability. Bugs in software that changes these configurations, such as installers, only worsen the situation. A self-managing configuration system should be continuously monitoring itself for invalid settings, preventing the bugs from harming the system. Unfortunately, while there are many constraints which can differentiate between valid and invalid settings, few are explicitly written down, much less written down in a form usable by an automatic monitor. We propose an approach to automatically infer these correctness constraints based on samples of known good configurations. In this paper we present Glean, a system for analyzing the structure of configurations and automatically inferring four types of correctness constraints on that structure.