D. Abbott

The Jefferson Lab trigger supervisor system

We discuss the design and performance of a trigger Supervisor System for use in nuclear physics experiments at Jefferson Lab. We also discuss the enhanced features of a new Trigger Supervisor Module now under construction.

Implementation of a level 1 trigger system using high speed serial (VXS) techniques for the 12GeV high luminosity experimental programs at Thomas Jefferson National Accelerator Facility

We will demonstrate a hardware and firmware solution for a complete fully pipelined multi-crate trigger system that takes advantage of the elegant high speed VXS serial extensions for VME. This trigger system includes three sections starting with the front end crate trigger processor (CTP), a global Sub-System Processor (SSP) and a Trigger Supervisor that manages the timing, synchronization and front end event readout. Within a front end crate, trigger information is gathered from each 16 Channel, 12bit Flash ADC module at 4nS intervals via the VXS backplane, to a Crate Trigger Processor (CTP). Each Crate Trigger Processor receives these 500MB/S VXS links from the 16 FADC-250 modules, aligns skewed data inherent of Aurora protocol, and performs real time crate level trigger algorithms. The algorithm results are encoded using a Reed-Solomon technique and transmission of this Level 1 trigger data is sent to the SSP using a multi-fiber link. The multi-fiber link achieves an aggregate trigger data transfer rate to the global trigger at 8Gb/s. The SSP receives and decodes Reed-Solomon error correcting transmission from each crate, aligns the data, and performs the global level trigger algorithms. The entire trigger system is synchronous and operates at 250MHz with the Trigger Supervisor managing not only the front end event readout, but also the distribution of the critical timing clocks, synchronization signals, and the global trigger signals to each front end readout crate. These signals are distributed to the front end crates on a separate fiber link and each crate is synchronized using a unique encoding scheme to guarantee that each front end crate is synchronous with a fixed latency, independent of the distance between each crate. The overall trigger signal latency is ≪3uS, and the proposed 12GeV experiments at Jefferson Lab require up to 200KHz Level 1 trigger rate.

CODA performance in the real world

The most ambitious implementation of the Jefferson Lab data acquisition system (CODA) to date is for the CLAS spectrometer in Experimental Hall B. CLAS has over 40,000 instrumented channels and uses up to 30 front-end (FASTBUS/VME) crates in the DAQ subsystem. During the initial experiments we found that performance of the fully instrumented DAQ system did not scale as expected based on single point to point benchmarks. Over the past year we have been able to study various performance bottlenecks in the CLAS DAQ system including front-end real time performance, switched 100BaseT Ethernet data transport, and online data distribution and recording. Performance tuning was necessary for components on both real time (VxWorks) and UNIX (Solaris) operating systems. In addition, a new efficient Event Transfer System (ET) was developed to provide faster online monitoring while having minimal impact on data throughput to storage. We discuss these issues and efforts to overcome the real world problems associated with running a high performance DAQ system on a variety of commercial hardware and software.

A 250 MHz Level 1 Trigger and Distribution System for the GlueX experiment

The GlueX detector now under construction at Jefferson Lab will search for exotic mesons though photoproduction (10^8 tagged photons per second) on a liquid hydrogen target. A Level 1 hardware trigger design is being developed to reduce total electromagnetic (≪ 200 MHz) and hadronic (≫ 350 kHz) rates to less than 200 kHz. This trigger is deadtimeless and operates on a global synchronized 250 MHz clock. The core of the trigger design is based on a custom pipelined flash ADC board that uses a VXS backplane to collect samples from all ADCs in a VME crate. A custom switch-slot board called a Crate Trigger Processor (CTP) processes this data and passes the crate level data via a multi-lane fiber optic link to the Global Trigger Processing Crate (also VXS). Within this crate detector sub-system processor (SSP) boards can accept all individual crate links. The subsystem data are processed and finally passed to global trigger boards (GTP) where the final L1 decision is made. We present details of the trigger design and report some performance results on current prototype systems

Jefferson Lab Data Acquisition Run Control System

A general overview of the Jefferson Lab data acquisition run control system is presented. This run control system is designed to operate the configuration, control, and monitoring of all Jefferson Lab experiments. It controls data-taking activities by coordinating the operation of DAQ sub-systems, online software components and third-party software such as external slow control systems. The main, unique feature which sets this system apart from conventional systems is its incorporation of intelligent agent concepts. Intelligent agents are autonomous programs which interact with each other through certain protocols on a peer-to-peer level. In this case, the protocols and standards used come from the domain-independent Foundation for Intelligent Physical Agents (FIPA), and the implementation used is the Java Agent Development Framework (JADE). A lightweight, XML/RDF-based language was developed to standardize the description of the run control system for configuration purposes.

Design of the Trigger Interface and Distribution board for CEBAF 12 GeV Upgrade

The design of the Trigger Interface and Distribution (TID) [1] board for the 12 GeV Upgrade [2] at the Continues Electron Beam Accelerator Facility (CEBAF) in TJNAL is described. The TID board distributes a low jitter system clock, synchronized trigger, and synchronized multi-purpose SYNC signal. The TID also initiates data acquisition for the crate. With the TID boards, a multi-crate system can be setup for experiment test and commissioning.

cMsg - a publish/subscribe package for real-time and online control systems

cMsg is a messaging system designed for use in realtime data acquisition and online controls systems, and includes support for C/C++ and Java clients. It is highly customizable and extensible, and includes a full-featured asynchronous publish/subscribe component as well as a synchronous peer-to-peer component. We first describe the publish/subscribe and peer-to-peer messaging paradigms, discuss their use in real-time and online systems, and describe the power and flexibility of the publish/subscribe paradigm. Next we describe the philosophy of the cMsg implementation and present some details and benchmarks. Finally, since the core of cMsg is written in pure Java, we discuss the suitability of Java for use in real-time and online systems

Evaluation of JAVA 2 and CORBA for platform independent control and monitoring of the next generation of the CODA data acquisition system

The CODA data acquisition system was developed as a common data acquisition environment for experiments at Jefferson lab. The control and monitoring system was developed in EIFFEL and later ported to C++. The original design ran on ULTRIX and has been ported to HP-UX, Solaris, IRIX and Linux. The porting process is tedious and error prone. The code is difficult to maintain and the principal authors have left the group. With this in mind we are evaluating JAVA as a portable, and stable, language with which to write the next generation run control system. In parallel we are evaluating CORBA as the communication mechanism. The recent release of Java 2 with integrated support for CORBA has made this route even more interesting. This paper presents the results of our evaluation of tools and technologies, experience gained while writing a prototype of the preliminary design for the control system.

Real-time performance of commercial Intel-based VME Controllers for the CODA data acquisition system

We have evaluated the performance of several Intel-based VME Controllers for the use in the data acquisition systems (DAQs) at Jefferson Lab. For the 12 GeV upgrade, PPC-based VME controllers running vxWorks will be replaced with those that are Intel-based running Linux. Their task will be facilitated by the use of FPGAs on the VME modules to perform trigger logic and communicating trigger information over serial and fiber connections through the DAQ. The need for a hard realtime operating system on the VME controller is removed from the equation as the readout of the digitized data from the VME modules (using VME-2eSST) is done in a threaded environment with multiple CPU cores while digitization takes place in the buffered, pipelined system. In this paper we briefly discuss a bench setup for evaluating a VME Controller and its kernel and user space environment. We present results from baseline testing of various models from different vendors using different Linux kernels, including results from a kernel compiled with the CONFIG_PREEMPT_RT patch.

cMsg - A general purpose, publish-subscribe, interprocess communication implementation and framework

cMsg is software used to send and receive messages in the Jefferson Lab online and runcontrol systems. It was created to replace the several IPC software packages in use with a single API. cMsg is asynchronous in nature, running a callback for each message received. However, it also includes synchronous routines for convenience. On the framework level, cMsg is a thin API layer in Java, C, or C++ that can be used to wrap most message-based interprocess communication protocols. The top layer of cMsg uses this same API and multiplexes user calls to one of many such wrapped protocols (or domains) based on a URL-like string which we call a Uniform Domain Locator or UDL. One such domain is a complete implementation of a publish-subscribe messaging system using network communications and written in Java (user APIs in C and C++ too). This domain is built in a way which allows it to be used as a proxy server to other domains (protocols). Performance is excellent allowing the system not only to be used for messaging but also as a data distribution system