Robert P. Sherman

Self-similarity through high-variability: statistical analysis of Ethernet LAN traffic at the source level

A number of empirical studies of traffic measurements from a variety of working packet networks have demonstrated that actual network traffic is self-similar or long-range dependent in nature-in sharp contrast to commonly made traffic modeling assumptions. We provide a plausible physical explanation for the occurrence of self-similarity in local-area network (LAN) traffic. Our explanation is based on convergence results for processes that exhibit high variability and is supported by detailed statistical analyzes of real-time traffic measurements from Ethernet LANs at the level of individual sources. This paper is an extended version of Willinger et al. (1995). We develop here the mathematical results concerning the superposition of strictly alternating ON/OFF sources. Our key mathematical result states that the superposition of many ON/OFF sources (also known as packet-trains) with strictly alternating ON- and OFF-periods and whose ON-periods or OFF-periods exhibit the Noah effect produces aggregate network traffic that exhibits the Joseph effect. There is, moreover, a simple relation between the parameters describing the intensities of the Noah effect (high variability) and the Joseph effect (self-similarity). An extensive statistical analysis of high time-resolution Ethernet LAN traffic traces confirms that the data at the level of individual sources or source-destination pairs are consistent with the Noah effect. We also discuss implications of this simple physical explanation for the presence of self-similar traffic patterns in modern high-speed network traffic.

Long-range dependence in variable-bit-rate video traffic

We analyze 20 large sets of actual variable-bit-rate (VBR) video data, generated by a variety of different codecs and representing a wide range of different scenes. Performing extensive statistical and graphical tests, our main conclusion is that long-range dependence is an inherent feature of VBR video traffic, i.e., a feature that is independent of scene (e.g., video phone, video conference, motion picture video) and codec. In particular, we show that the long-range dependence property allows us to clearly distinguish between our measured data and traffic generated by VBR source models currently used in the literature. These findings give rise to novel and challenging problems in traffic engineering for high-speed networks and open up new areas of research in queueing and performance analysis involving long-range dependent traffic models. A small number of analytic queueing results already exist, and we discuss their implications for network design and network control strategies in the presence of long-range dependent traffic. >

Proof of a fundamental result in self-similar traffic modeling

We state and prove the following key mathematical result in self-similar traffic modeling: the superposition of many ON/OFF sources (also known as packet trains) with strictly alternating ON- and OFF-periods and whose ON-periods or OFF-periods exhibit the Noah Effect (i.e., have high variability or infinite variance) can produce aggregate network traffic that exhibits the Joseph Effect (i.e., is self-similar or long-range dependent). There is, moreover, a simple relation between the parameters describing the intensities of the Noah Effect (high variability) and the Joseph Effect (self-similarity). This provides a simple physical explanation for the presence of self-similar traffic patterns in modern high-speed network traffic that is consistent with traffic measurements at the source level. We illustrate how this mathematical result can be combined with modern high-performance computing capabilities to yield a simple and efficient linear-time algorithm for generating self-similar traffic traces.We also show how to obtain in the limit a Lévy stable motion, that is, a process with stationary and independent increments but with infinite variance marginals. While we have presently no empirical evidence that such a limit is consistent with measured network traffic, the result might prove relevant for some future networking scenarios.

Self-similarity through high-variability: statistical analysis of ethernet LAN traffic at the source level

A number of recent empirical studies of traffic measurements from a variety of working packet networks have convincingly demonstrated that actual network traffic is self-similar or long-range dependent in nature (i.e., bursty over a wide range of time scales) - in sharp contrast to commonly made traffic modeling assumptions. In this paper, we provide a plausible physical explanation for the occurrence of self-similarity in high-speed network traffic. Our explanation is based on convergence results for processes that exhibit high variability (i.e., infinite variance) and is supported by detailed statistical analyses of real-time traffic measurements from Ethernet LANs at the level of individual sources.Our key mathematical result states that the superposition of many ON/OFF sources (also known as packet trains) whose ON-periods and OFF-periods exhibit the Noah Effect (i.e., have high variability or infinite variance) produces aggregate network traffic that features the Joseph Effect (i.e., is self-similar or long-range dependent). There is, moreover, a simple relation between the parameters describing the intensities of the Noah Effect (high variability) and the Joseph Effect (self-similarity). An extensive statistical analysis of two sets of high time-resolution traffic measurements from two Ethernet LANs (involving a few hundred active source-destination pairs) confirms that the data at the level of individual sources or source-destination pairs are consistent with the Noah Effect. We also discuss implications of this simple physical explanation for the presence of self-similar traffic patterns in modern high-speed network traffic for (i) parsimonious traffic modeling (ii) efficient synthetic generation of realistic traffic patterns, and (iii) relevant network performance and protocol analysis.

The limiting distribution of the maximum rank correlation estimator

Hans maximum rank correlation estimator is shown to be square-root n-consistent and asymptotically normal. The proof rests on a general method for determining the asymptotic distribution of a maximization estimator, a simple U-statistic decomposition, and a uniform bound for degenerate U-processes. A consistent estimator of the asymptotic covariance matrix is provided, along with a result giving the explicit form of this matrix for any model within the scope of the maximum rate correlation estimator. The latter result is applied to the binary choice model, and it is found that the maximum rate correlation estimator does not achieve the semiparametric efficiency bound. Copyright 1993 by The Econometric Society.

Rank estimators for monotonic index models

We present a new class of rank estimators of scaled coefficients in semiparametric monotonic linear index models. The estimators require no subjective bandwidth choices and have attractive computational properties. We establish √n-consistency and asymptotic normality, and provide the general form and consistent estimators of the asymptotic covariance matrix. We also provide a generalization covering single equation multiple-indices models satisfying certain monotonicity constraints. An analogue of consistency when all explanatory variables are categorical is established, and an application is presented.

U-Processes in the Analysis of a Generalized Semiparametric Regression Estimator

We prove null-consistency and asymptotic normality of a generalized semiparametric regression estimator that includes as special cases Ichimuras semiparametric least-squares estimator for single index models, and the estimator of Klein and Spady for the binary choice regression model. Two function expansions reveal a type of U-process structure in the criterion function; then new U-process maximal inequalities are applied to establish the requisite stochastic equicontinuity condition. This method of proof avoids much of the technical detail required by more traditional methods of analysis. The general framework suggests other null-consistent and asymptotically normal estimators.

Estimating new product demand from biased survey data

Abstract Market researchers often conduct surveys asking respondents to estimate their future demand for new products. However, projected demand may exhibit systematic bias. For example, the more respondents like a product, the more they may exaggerate their demand. We found evidence of such exaggeration in a recent survey of demand for a potential new video product. In this paper, we develop a computationally tractable procedure that corrects for a general form of systematic bias in demand projections. This general form is characterized by a monotonictransformation of projected demand, and covers exaggeration bias as a special case.

TESTS OF CERTAIN TYPES OF IGNORABLE NONRESPONSE IN SURVEYS SUBJECT TO ITEM NONRESPONSE OR ATTRITION

Analysts of cross-sectional or panel surveys often base inferences about relationships between variables on complete units, excluding units that are incomplete due to item nonresponse or attrition. This practice is justiable if exclusion is ignorable in an appropriate sense. This paper characterizes certain types of ignorable exclusion in surveys subject to item nonresponse and develops tests based on these characterizations. These tests are applied to data from several National Election Study panels and evidence is found of violations of these characterizations. Characterizations and tests of certain types of ignorable attrition in standard panel surveys are also presented.

Some Convergence Theory For Iterative Estimation Procedures With An Application To Semiparametric Estimation

We develop general conditions for rates of convergence and convergence in distribution of iterative procedures for estimating finite-dimensional parameters. An asymptotic contraction mapping condition is the centerpiece of the theory. We illustrate some of the results by deriving the limiting distribution of a two-stage iterative estimator of regression parameters in a semiparametric binary response model. Simulation results illustrating the computational benefits of the first-stage iterative estimator are also reported.We thank a co-editor and two referees for comments and criticisms that led to significant improvements in this paper. We also thank Roger Klein for providing us with Gauss code to compute his estimator.