Kingsum Chow

SPECjvm2008 Performance Characterization

SPECjvm2008 is a new multi-threaded Java benchmark from SPEC and it replaces the aging single threaded SPECjvm98. The benchmark is intended to address several shortcomings of the earlier workloads in SPECjvm98 by replacing DB, Chart, Javac; removing Jess, adding XML, Serial, Crypto, in-cache and out-cache versions of Scimark workloads. It is targeted for measuring the performance of both JVM and hardware systems. In this paper we describe the salient features of SPECjvm2008. We then take a first look at the performance of this benchmark on current multi-core platforms and study the sensitivity of the components of the workload to basic architectural aspects such as the number of processor cores, the processor frequency, cache and memory sub-system. We focus our study on understanding how the behavior of this workload compares with other standard Java benchmarks, SPECjbb2005 and SPECjAppServer2004, both in components of the software stack that the workloads touch as well as in the aspects of the platform that they exercise and draw conclusion on the usefulness of SPECjvm2008 for practitioners of JVM and hardware performance analysis.

Constructing a Non-Linear Model with Neural Networks for Workload Characterization

Workload characterization involves the understanding of the relationship between workload configurations and performance characteristics. To better assess the complexity of workload behavior, a model based approach is needed. Nevertheless, several configuration parameters and performance characteristics exhibit non-linear relationships that prohibit the development of an accurate application behavior model. In this paper, we propose a non-linear model based on an artificial neural network to explore such complex relationship. We achieved high accuracy and good predictability between configurations and performance characteristics when applying such a model to a 3-tier setup with response time restrictions. As shown by our work, a non-linear model and neural networks can increase the understandings of complex multi-tiered workloads, which further provide useful insights for performance engineers to tune their workloads for improving performance

Sandbox Prefetching: Safe run-time evaluation of aggressive prefetchers

Memory latency is a major factor in limiting CPU performance, and prefetching is a well-known method for hiding memory latency. Overly aggressive prefetching can waste scarce resources such as memory bandwidth and cache capacity, limiting or even hurting performance. It is therefore important to employ prefetching mechanisms that use these resources prudently, while still prefetching required data in a timely manner. In this work, we propose a new mechanism to determine at run-time the appropriate prefetching mechanism for the currently executing program, called Sandbox Prefetching. Sandbox Prefetching evaluates simple, aggressive offset prefetchers at run-time by adding the prefetch address to a Bloom filter, rather than actually fetching the data into the cache. Subsequent cache accesses are tested against the contents of the Bloom filter to see if the aggressive prefetcher under evaluation could have accurately prefetched the data, while simultaneously testing for the existence of prefetchable streams. Real prefetches are performed when the accuracy of evaluated prefetchers exceeds a threshold. This method combines the ideas of global pattern confirmation and immediate prefetching action to achieve high performance. Sandbox Prefetching improves performance across the tested workloads by 47.6% compared to not using any prefetching, and by 18.7% compared to the Feedback Directed Prefetching technique. Performance is also improved by 1.4% compared to the Access Map Pattern Matching Prefetcher, while incurring considerably less logic and storage overheads.

Detecting resource leaks through dynamical mining of resource usage patterns

Resource management is crucial to software productions. Resources must be carefully acquired and released, or a resource leak might occur. For open source projects, resource leaks can be easily introduced during code check-in, and it is laborious to review, identify, report, and fix such leaks. Recently, there has been a growing interest in data mining API usage patterns to discover potential bugs such as resource leaks. However, the usage patterns mined are specific to a certain library, which cannot be applied to detect bugs in other libraries. In this paper, we present an idea called MODE, “Mine Once, Detect Everywhere”, to address the universality of such patterns, and use them to detect potential resource leaks automatically before code check-in. We propose an efficient algorithm to record the most valuable API calls that are related to resource usage during program execution, and mine resource usage patterns from the traces with a sequence miner. To verify the effectiveness of the patterns, experiments are given to use them to detect real resource leaks in large open source projects.

Hierarchical Means: Single Number Benchmarking with Workload Cluster Analysis

Benchmark suite scores are typically calculated by averaging the performance of each individual workload. The scores are inherently affected by the distribution of workloads. Given the applications of a benchmark suite are typically contributed by many consortium members, workload redundancy becomes inevitable. Especially, the merger of the benchmarks can significantly increase artificial redundancy. Redundancy in the workloads of a benchmark suite renders the benchmark scores biased, making the score of a suite susceptible to malicious tweaks. The current standard workaround method to alleviating the redundancy issue is to weigh each individual workload during the final score calculation. Unfortunately, such a weight-based score adjustment can significantly undermine the credibility of the objectiveness of benchmark scores. In this paper, we propose a set of benchmark suite score calculation methods called the hierarchical means that incorporate cluster analysis to amortize the negative effect of workload redundancy. These methods not only improve the accuracy and robustness of the score, but also improve the objectiveness over the weight-based approach. In addition, they can also be used to analyze the inherent redundancy and cluster characteristics in a quantitative manner for evaluating a new benchmark suite. In our case study, the hierarchical geometric mean was applied to a hypothetical Java benchmark suite, which attempts to model the upcoming release of the new SPECjvm benchmark suite. In addition, we also show that benchmark suite clustering heavily depends on how the workloads are characterized.

Characterization of Java/sup TM/ application server workloads

This paper examines the workload characterization of a Java 2 Enterprise Edition (J2EE/sup TM/) application server workload. The application provides services for a mixture of e-commerce transactions continuously. This paper examines the variation of system observable behavior, program behavior and performance characteristics at the CPU level and their impact as a result of the variation of the payload of those different kinds of e-commerce transactions. Early results indicate that some methods could be useful in establishing the relationship between observable system configurations and program behavior. Furthermore, performance at the CPU level may be linked to these configurations. It may also help computer architects in collecting traces of workload that can cover a wide spectrum for performance simulation and analysis.

Resource Trade-Offs for Java Applications in the Cloud

Java applications form an important class of applications running in the data center and in the cloud. They may perform better when more memory can be used in the heap, as the time spent in garbage collections is reduced. However, when ample CPU is available and memory is tight, such Java applications may do well with a smaller heap as it can absorb the cost of more garbage collections. In the cloud, the amount of resources available may vary from time to time. This paper investigates an approach based on the statistical design of experiments and performance data analytics to make resource trade-offs, between CPU and memory, to increase datacenter efficiency in the cloud.