Joseph Horowitz

Multicast-based inference of network-internal loss characteristics

Robust measurements of network dynamics are increasingly important to the design and operation of large internetworks like the Internet. However, administrative diversity makes it impractical to monitor every link on an end-to-end path. At the same time, it is difficult to determine the performance characteristics of individual links from end-to-end measurements of unicast traffic. In this paper, we introduce the use of end-to-end measurements of multicast traffic to infer network-internal characteristics. The bandwidth efficiency of multicast traffic makes it suitable for large-scale measurements of both end-to-end and internal network dynamics. We develop a maximum-likelihood estimator for loss rates on internal links based on losses observed by multicast receivers. It exploits the inherent correlation between such observations to infer the performance of paths between branch points in the tree spanning a multicast source and its receivers. We derive its rate of convergence as the number of measurements increases, and we establish robustness with respect to certain generalizations of the underlying model. We validate these techniques through simulation and discuss possible extensions and applications of this work

Multicast-based inference of network-internal delay distributions

Packet delay greatly influences the overall performance of network applications. It is therefore important to identify causes and locations of delay performance degradation within a network. Existing techniques, largely based on end-to-end delay measurements of unicast traffic, are well suited to monitor and characterize the behavior of particular end-to-end paths. Within these approaches, however, it is not clear how to apportion the variable component of end-to-end delay as queueing delay at each link along a path. Moreover, there are issues of scalability for large networks.In this paper, we show how end-to-end measurements of multicast traffic can be used to infer the packet delay distribution and utilization on each link of a logical multicast tree. The idea, recently introduced in [3] and [4], is to exploit the inherent correlation between multicast observations to infer performance of paths between branch points in a tree spanning a multicast source and its receivers. The method does not depend on cooperation from intervening network elements; because of the bandwidth efficiency of multicast traffic, it is suitable for large-scale measurements of both end-to-end and internal network dynamics. We establish desirable statistical properties of the estimator, namely consistency and asymptotic normality. We evaluate the estimator through simulation and observe that it is robust with respect to moderate violations of the underlying model.

Multicast topology inference from measured end-to-end loss

The use of multicast inference on end-to-end measurement has been proposed as a means to infer network internal characteristics such as packet link loss rate and delay. We propose three types of algorithm that use loss measurements to infer the underlying multicast topology: (i) a grouping estimator that exploits the monotonicity of loss rates with increasing path length; (ii) a maximum-likelihood estimator (MLE); and (iii) a Bayesian estimator. We establish their consistency, compare their complexity and accuracy, and analyze the modes of failure and their asymptotic probabilities.

The use of end-to-end multicast measurements for characterizing internal network behavior

We present a novel methodology for identifying internal network performance characteristics based on end-to-end multicast measurements. The methodology, solidly grounded on statistical estimation theory, can be used to characterize the internal loss and delay behavior of a network. Measurements on the MBone have been used to validate the approach in the case of losses. Extensive simulation experiments provide further validation of the approach, not only for losses, but also for delays. We also describe our strategy for deploying the methodology on the Internet. This includes the continued development of the National Internet Measurement Infrastructure to support RTP-based end-to-end multicast measurements and the development of software tools to analyze the traces. Once complete, this combined software/hardware infrastructure will provide a service for understanding and forecasting the performance of the Internet.

Multicast-based inference of network-internal characteristics: accuracy of packet loss estimation

We explore the use of end-to-end multicast traffic as measurement probes to infer network internal characteristics. We have developed in an earlier paper a maximum likelihood estimator for packet loss rates on individual links based on losses observed by multicast receivers. This technique exploits the inherent correlation between such observations to infer the performance of paths between branch points in the multicast tree spanning the probe source and its receivers. We evaluate through analysis and simulation the accuracy of our estimator under a variety of network conditions. In particular, we report on the error between inferred loss rates and actual loss rates as we vary the network topology, propagation delay, packet drop policy, background traffic mix, and probe traffic type. In all but one case, estimated losses and probe losses agree to within 2 percent on average. We feel this accuracy is enough to reliably identify congested links in a wide-area internetwork.

Network Delay Tomography from End-to-End Unicast Measurements

In this paper, we explore the use of end-to-end unicast traffic measurements to estimate the delay characteristics of internal network links. Experiments consist of back-to-back packets sent from a sender to pairs of receivers. Building on recent work [11,5,4], we develop efficient techniques for estimating the link delay distribution. Moreover, we also provide a method to directly estimate the link delay variance, which can be extended to the estimation of higher order cumulants. Accuracy of the proposed techniques depends on strong correlation between the delay seen by the two packets along the shared path. We verify the degree of correlation in packet pairs through network measurements. We also use simulation to explore the performance of the estimator in practice and observe good accuracy of the inference techniques.

Multicast-based loss inference with missing data

Network tomography using multicast probes enables inference of loss characteristics of internal network links from reports of end-to-end loss seen at multicast receivers. We develop estimators for internal loss rates when reports are not available on all probes or from all receivers. This problem is motivated by the use of unreliable transport protocols, such as reliable transport protocol, to transmit loss reports to a collector for inference. We use a maximum-likelihood (ML) approach in which we apply the expectation maximization (EM) algorithm to provide an approximating solution to the the ML estimator for the incomplete data problem. We present a concrete realization of the algorithm that can be applied to measured data. For classes of models, we establish identifiability of the probe and report loss parameters, and convergence of the EM sequence to the maximum-likelihood estimator (MLE). Numerical results suggest that these properties hold more generally. We derive convergence rates for the EM iterates, and the estimation error of the MLE. Finally, we evaluate the accuracy and convergence rate through extensive simulations.

Loss-based inference of multicast network topology

The use of multicast inference on end-to-end measurement has recently been proposed as a means to infer network internal characteristics such as packet loss rate and network topology. In this paper we propose and evaluate new algorithms for multicast topology inference based on measurement of end-to-end loss. We compare their accuracy and comment on their computational complexity.

Mean rates of convergence of empirical measures in the Wasserstein metric

Abstract An upper bound is given for the mean square Wasserstein distance between the empirical measure of a sequence of i.i.d. random vectors and the common probability law of the sequence. The same result holds for an infinite exchangeable sequence and its directing measure. Similarly, for an i.i.d. sequence of stochastic processes, an upper bound is obtained for the mean square of the maximum, over 0 ⩽ t ⩽ T, of the Wasserstein distance between the empirical measure of the sequence at time t and the common marginal law at t. These upper bounds are derived under weak assumptions and are not very far from the known rate of convergence pertaining to an i.i.d. sequence of uniform random vectors on the unit cube. Our approach, however, allows us to get results for arbitrary distributions under moment conditions and also gives results for processes. An application is given to so-called diffusions with jumps. Moment estimates for these processes are derived which may be of independent interest.

Probabilistic Model of Microbial Cell Growth, Division, and Mortality

ABSTRACT After a short time interval of length δt during microbial growth, an individual cell can be found to be divided with probability Pd(t)δt, dead with probability Pm(t)δt, or alive but undivided with the probability 1 − [Pd(t) + Pm(t)]δt, where t is time, Pd(t) expresses the probability of division for an individual cell per unit of time, and Pm(t) expresses the probability of mortality per unit of time. These probabilities may change with the state of the population and the habitats properties and are therefore functions of time. This scenario translates into a model that is presented in stochastic and deterministic versions. The first, a stochastic process model, monitors the fates of individual cells and determines cell numbers. It is particularly suitable for small populations such as those that may exist in the case of casual contamination of a food by a pathogen. The second, which can be regarded as a large-population limit of the stochastic model, is a continuous mathematical expression that describes the populations size as a function of time. It is suitable for large microbial populations such as those present in unprocessed foods. Exponential or logistic growth with or without lag, inactivation with or without a “shoulder,” and transitions between growth and inactivation are all manifestations of the underlying probability structure of the model. With temperature-dependent parameters, the model can be used to simulate nonisothermal growth and inactivation patterns. The same concept applies to other factors that promote or inhibit microorganisms, such as pH and the presence of antimicrobials, etc. With Pd(t) and Pm(t) in the form of logistic functions, the model can simulate all commonly observed growth/mortality patterns. Estimates of the changing probability parameters can be obtained with both the stochastic and deterministic versions of the model, as demonstrated with simulated data.