John Klein

A systematic review of system-of-systems architecture research

Context: A system of systems is an assemblage of components which individually may be regarded as systems, and which possesses the additional properties that the constituent systems are operationally independent, and are managerially independent. Much has been published about the field of systems of systems by researchers and practitioners, often with the assertion that the system-of-systems design context necessitates the use of architecture approaches that are somewhat different from system-level architecture. However, no systematic review has been conducted to provide an extensive overview of system of systems architecture research. Objective: This paper presents such a systematic review. The objective of this review is to classify and provide a thematic analysis of the reported results in system of systems architecture. Method: The primary studies for the systematic review were identified using a predefined search strategy followed by an extensive manual selection process. Results: We found the primary studies published in a large number of venues, mostly domain-oriented, with no obvious center of a research community of practice. The field seems to be maturing more slowly than other software technologies: Most reported results described individuals or teams working in apparent isolation to develop solutions to particular system-of-systems architecture problems, with no techniques gaining widespread adoption. Conclusions: A comprehensive research agenda for this field should be developed, and further studies should be performed to determine whether the information system-related problems of system of systems architecture are covered by existing software architecture knowledge, and if not, to develop general methods for system-of-systems architecture.

Performance Evaluation of NoSQL Databases: A Case Study

The choice of a particular NoSQL database imposes a specific distributed software architecture and data model, and is a major determinant of the overall system throughput. NoSQL database performance is in turn strongly influenced by how well the data model and query capabilities fit the application use cases, and so system-specific testing and characterization is required. This paper presents a method and the results of a study that selected among three NoSQL databases for a large, distributed healthcare organization. While the method and study considered consistency, availability, and partition tolerance (CAP) tradeoffs, and other quality attributes that influence the selection decision, this paper reports on the performance evaluation method and results. In our testing, a typical workload and configuration produced throughput that varied from 225 to 3200 operations per second between database products, while read operation latency varied by a factor of 5 and write latency by a factor of 4 (with the highest throughput product delivering the highest latency). We also found that achieving strong consistency reduced throughput by 10-25% compared to eventual consistency.

A reference architecture for big data systems in the national security domain

Acquirers, system builders, and other stakeholders of big data systems need to define requirements, develop and evaluate solutions, and integrate systems together. A reference architecture enables these software engineering activities by standardizing nomenclature, defining key solution elements and their relationships, collecting relevant solution patterns, and classifying existing technologies. Within the national security domain, existing reference architectures for big data systems have not been useful because they are too general or are not vendor-neutral. We present a reference architecture for big data systems that is focused on addressing typical national defence requirements and that is vendor-neutral, and we demonstrate how to use this reference architecture to define solutions in one mission area.

Architecture Knowledge for Evaluating Scalable Databases

Designing massively scalable, highly available big data systems is an immense challenge for software architects. Big data applications require distributed systems design principles to create scalable solutions, and the selection and adoption of open source and commercial technologies that can provide the required quality attributes. In big data systems, the data management layer presents unique engineering problems, arising from the proliferation of new data models and distributed technologies for building scalable, available data stores. Architects must consequently compare candidate database technology features and select platforms that can satisfy application quality and cost requirements. In practice, the inevitable absence of up-to-date, reliable technology evaluation sources makes this comparison exercise a highly exploratory, unstructured task. To address these problems, we have created a detailed feature taxonomy that enables rigorous comparison and evaluation of distributed database platforms. The taxonomy captures the major architectural characteristics of distributed databases, including data model and query capabilities. In this paper we present the major elements of the feature taxonomy, and demonstrate its utility by populating the taxonomy for nine different database technologies. We also briefly describe QuABaseBD, a knowledge base that we have built to support the population and querying of database features by software architects. QuABaseBD links the taxonomy to general quality attribute scenarios and design tactics for big data systems. This creates a unique, dynamic knowledge resource for architects building big data systems.

Understanding patterns for system of systems integration

Architecting systems of systems is well known to be a formidable challenge. A major aspect in this is defining the integration among the systems that constitute the system of systems. In this paper, we aim to support the SoS architect by systematically developing a way to characterize system of systems integration patterns. These characteristics at the same time support the architecting process by highlighting important issues a SoS architect needs to consider. We discuss the consolidated template and illustrate it with an example pattern. We also discuss the integration of this novel pattern approach into the overall architecting process.

Model-Driven Observability for Big Data Storage

The scale, heterogeneity, and pace of evolution of the storage components in big data systems makes it impractical to manually insert monitoring code for observability metric collection and aggregation. In this paper we present an architecture that automates these metric collection processes, using a model-driven approach to configure a distributed runtime observability framework. We describe and evaluate an implementation of the architecture that collects and aggregates metrics for a big data system using heterogeneous NoSQL data stores. Our scalability tests demonstrate that the implementation can monitor 20 different metrics from 10,000 database nodes with a sampling interval of 20 seconds. Below this interval, we lose metrics due to the sustained write load required in the metrics database. This indicates that observability at scale must be able to support very high write loads in a metrics collection database.

What Makes an Architect Successful

An architect whose skills and capabilities match a projects needs will more likely be successful. Moreover, each software life-cycle phase requires different skills. A proposed model identifies the skills needed at each phase and helps explain common failure patterns.

An early look at defining variability requirements for system of systems platforms

In the commercial domain, platform-based approaches, in which a set of functions or services are bundled to form the basis of many products, have enabled efficient development of systems and their composition into systems of systems. A successful platform must balance sufficient commonality to support economical reuse, while also providing variability and extensibility to enable innovation in system and system of systems (SoS) capabilities. These commonality/variability tradeoffs for SoS platforms are frequently tacit decisions, since there are no accepted techniques for analyzing such decisions at the scale and degree of requirements uncertainty that characterize most SoSs. The objective of our work is to develop a method for analyzing decisions about requirements for common platforms for SoSs. The method begins with the requirements tasks of identifying and selecting appropriate variabilities (variation points, variation ranges, and variation decision binding times) to support immediate SoS needs, and also enable innovation and controlled evolution. We are currently conducting a workshop and interviews with SoS experts to define the essential technical problems in SoS common platform development and identify solution constraints. We will then define a simplified SoS with limited capability requirements to use as a model problem. We will use the model problem to assess the fit of existing scope, commonality, and variability methods from software product lines to the SoS context, and extend existing economic models using real options and probabilistic models to model uncertainty in evolution requirements. While it is too early to draw firm conclusions about the effectiveness of our approach, it is based on proven technologies from the mature field of software product lines and so we have confidence that we can build successful SoS techniques from this foundation.

System-of-systems platform scoping

In a system of systems, interoperability among constituent systems is a primary architecture concern. One architecture strategy to promote interoperability in this design context is to create a system-of-systems platform to provide certain common services to all constituent systems, with the goal of reducing development effort for the constituent systems and reducing integration and assurance cost for the system of systems. A successful system-of-systems platform must balance sufficient commonality to support economical reuse, while also providing variability and extensibility to enable innovation in system and system of systems capabilities. These commonality/variability tradeoffs for system-of-systems platforms are frequently tacit decisions, since there are no existing techniques for analyzing such decisions at the scale and degree of requirements uncertainty that characterize most systems of systems.

Using Stakeholder Preferences to Make Better Architecture Decisions

A roadmap to modernize the architecture of an existing system must satisfy many strongly-positioned stakeholders and satisfy the constraints of continuing business operations as the plan is implemented. Our previous work reported on a method to engage with stakeholders to model architecture options for a modernization roadmap. These models have proven to be too large to analyze all options manually: Ad hoc approaches must be employed to prune the space of possible solutions, which risks dropping optimal solutions. We report here on a method that efficiently collects stakeholder preferences about architecture options and uses an automated search-based optimization approach over the full solution space to identify the most important architecture decisions, i.e., the decisions that have the most influence on stakeholder satisfaction.