Michael Ogg

NILE: wide-area computing for high energy physics

The CLEO project [2], centered at Cornell University, is alarge-scale high energy physics project. The goals of the projectarise from an esoteric question---why is there apparently so littleantimatter in the universe?---and the computational problems thatarise in trying to answer this question are quite challenging. To answer this question, the CESR storage ring at Cornell isused to generate a beam of electrons directed at an equally strongbeam of positrons. These two beams meet inside a detector that isembedded in a magnetic field and is equipped with sensors. Thecollisions of electrons and positrons generate several secondarysubatomic particles. Each collision is called an event andis sensed by detecting charged particles (via the ionization theyproduce in a drift chamber) and neutral particles (in the case ofphotons, via their deposition of energy in a crystal calorimeter),as well as by other specialized detector elements. Most events areignored, but some are recorded in what is called raw data(typically 8Kbytes per event). Offline, a second program calledpass2 computes, for each event, the physical properties ofthe particles, such as their momenta, masses, and charges. Thiscompute-bound program produces a new set of records describing theevents (now typically 20Kbytes per event). Finally, a third programreads these events, and produces a lossily-compressed version ofonly certain frequently-accessed fields, written in what is calledroar format (typically 2Kbytes per event). The physicists analyze this data with programs that are, for themost part, embarrassingly parallel and I/O limited. Such programstypically compute a result based on a projection of a selection ofa large number of events, where the result is insensitive to theorder in which the events are processed. For example, a program mayconstruct histograms, or compute statistics, or cull the rawdata for physical inspection. The projection is either the completepass2 record or (much more often) the smaller roarrecord, and the selection is done in an ad-hoc manner by theprogram itself. Other programs are run as well. For example, a Monte Carlosimulation of the experiment is also run (called montecarlo) in order to correct the data for detector acceptance andinefficiencies, as well as testing aspects of the model used tointerpret the data. This program is compute bound. Anotherimportant example is called recompress. Roughly every twoyears, improvements in detector calibration and reconstructionalgorithms make it worthwhile to recompute more accuratepass2 data (and hence, more accurate roar data) fromall of the raw data. This program is compute-bound (itcurrently requires 24 200-MIP workstations running flat out forthree months) and so must be carefully worked into the schedule sothat it does not seriously impact the ongoing operations. Making this more concrete, the current experiment generatesapproximately 1 terabyte of event data a year. Only recentroar data can be kept on disk; all other data must reside ontape. The data processing demands consume approximately 12,000SPECint92 cycles a year. Improvements in the performance of CESRand the sensitivity of the detector will cause both of these valuesto go up by a factor of ten in the next few years, which willcorrespondingly increase the storage and computational needs by afactor of ten. The CLEO project prides itself on being able to do big scienceon a tight budget, and so the programming environment that the CLEOproject provides for researchers is innovative but somewhatprimitive. Jobs that access the entire data set can take days tocomplete. To circumvent limited access to tape, the network, orcompute resources close to the central disk, physicists often dopreliminary selections and projections (called skims) tocreate private disk data sets of events for further local analysis.Limited resources usually exact a high human price for resource andjob management and ironically, can sometimes lead toinefficiencies. Given the increase in data storage, data retrieval,and computational needs, it has become clear that the CLEOphysicists require a better distributed environment in which to dotheir work. Hence, an NSF-funded National Challenge project was started withparticipants from both high energy physics, distributed computing,and data storage, in order to provide a better environment for theCLEO experiment. The goals of this project, called NILE [7], are: to build a scalable environment for storing and processing HighEnergy Physics data from the CLEO experiment. The environment mustscale to allow 100 terabytes or more of data to be addressable, andto be able to use several hundreds of geographically dispersedprocessors.to radically decrease the processing time of computationsthrough parallelism.to be practicable. NILE, albeit in a limited form,should be deployed very soon, and evolve to its full form by theend of the project in June 1999.Finally, the CLEO necessity of building on a budget carries overto NILE. There aresome more expensive resources, such as ATM switches and tape silos,that it will be necessary to use. However, as far as possible weare using commodity equipment, and free or inexpensive softwarewhenever possible. For example, one of our principal developmentplatforms is Pentium-based PCs, interconnected with 100 MbpsEthernet, running Linux and the GNU suite of tools.

The NILE System Architecture: Fault-Tolerant, Wide-Area Access to Computing and Data Resources

Nile is a multidisciplinary project building a distributed computing environment for HEP. It provides wide-area, fault-tolerant, integrated access to processing and data resources for collaborators of the CLEO experiment, though the goals and principles are applicable to many domains. Nile has three main objectives: a realistic distributed system architecture design, the design of a robust data model, and a Fast-Track implementation providing a prototype design environment which will also be used by CLEO physicists. This paper focuses on the software and wide-area system architecture design and the computing issues involved in making Nile services highly-available. 1 The Challenge and Goals The main goals of the Nile project are to build a scalable environment that gives access to a widely distributed set of resources for storing and processing HEP events, to increase the processing speed of computations, and to broaden access to event data so that analyses can be performed at geographically dispersed sites. The main obstacles in achieving these goals are the amount of data recorded by the CLEO experimentt1], and the data processing demands of the computing environment in the form of CPU cycles, network bandwidth and latency, and storage. To add to the complexity, diierent forms of computation impose diierent burdens on processing resources (either CPU bound or I/O bound). In the CLEO II experiment, a typical hadronic event is 8 kB, and grows to 20 kB when the results of event reconstruction are added to the event record. About 10 6 events are recorded each day, resulting in the production of about 1 TB of analyzed data per year. The actual amount of data transferred for each event during analysis depends upon the details of the Nile Data Modell3]. Approximately 3,000 SPECints of distributed CPU power per year are necessary for continuous event reconstruction on the incoming data. Another 7,000 SPECints are necessary for continuous Monte Carlo simulation of events, and 2,000 SPECints for analysis. Current plans for the improvement of the CESR storage ring and the CLEO III upgradee2] will increase these requirements by a factor of ten within four years.

An evaluation of PCs for high energy physics under Windows NT and Linux

The high availability and commodity prices of Intel-based PCs have made them serious contenders to workstations. We address the question whether PCs are appropriate for running real physics codes. We evaluate the performance of Intel-based systems with two popular 32-bit operating systems, Linux and Windows NT. We report on a suite of benchmark tests, both generic and HEP-specific, where we compare the performance of these two systems to each other, and to other popular workstation class machines.

Architecture decisions for wide area applications

A circuit which locks the signaling circuit of a telephone station. A microprocessor disconnects a keypad from an associated tone generator upon detection of operation of a lock button. The microprocessor connects the keypad to the tone generator upon detection of operation of the lock button and a predetermined unlock code provided by the keypad.

A middleware overview: goals, research issues, and coded examples (abstract)

In this tutorial, I will focus on the roles, types, and issues of middleware. I will give a short overview of the software problems that middleware addresses and (mostly) solves. I will do a survey of the different kinds of middleware and their applicability to diverse problem domains. I will spend somewhat longer discussing, with examples, CORBA, RMI, and Jini, in particular pointing out relative strengths and weaknesses. Finally, I will discuss some of the major and outstanding research issues, especially those that a PODC audience might find stimulating, and suggest where and how solutions might be found.

The NILE Data Model

Nile is a multi-disciplinary project building a distributed computing environment for HEP. Nile will provide fault-tolerant, integrated access to processing and data resources for collaborators of the CLEO experiment, though the goals and principles are applicable to many domains. Nile currently has three main objectives: a realistic distributed system architecture design, the design of a robust data model, and a Fast-Track implementation providing a prototype design environment to be used by CLEO physicists. In this paper, we describe the Data Model, its design issues, and its interactions with the Nile System Architecture.