G. Lamanna

A high-resolution TDC-based board for a fully digital Trigger and Data AcQuisition system in the NA62 experiment at CERN

A time-to-digital converter-based system, to be used for most subdetectors in the high-flux rare-decay experiment NA62 at CERN SPS, was built as part of the NA62 fully digital trigger and data acquisition system in which the TDC Board (TDCB) and a general-purpose motherboard (TEL62) will play a fundamental role. While TDCBs, housing four high-performance time-to-digital converters (HPTDCs), measure hit times from subdetectors, the motherboard processes and stores them in a buffer, produces trigger primitives from different detectors, and extracts only data related to the lowest trigger level decision, once this is taken on the basis of the trigger primitives themselves. The features of the TDCB developed by the Pisa NA62 group are extensively discussed and performance data are presented in order to show its compliance with the experiment requirements.

NaNet-10: A 10GbE network interface card for the GPU-based low-level trigger of the NA62 RICH detector

A GPU-based low level (L0) trigger is currently integrated in the experimental setup of the RICH detector of the NA62 experiment to assess the feasibility of building more refined physics-related trigger primitives and thus improve the trigger discriminating power. To ensure the real-time operation of the system, a dedicated data transport mechanism has been implemented: an FPGA-based Network Interface Card (NaNet-10) receives data from detectors and forwards them with low, predictable latency to the memory of the GPU performing the trigger algorithms. Results of the ring-shaped hit patterns reconstruction will be reported and discussed.

NaNet: a low-latency NIC enabling GPU-based, real-time low level trigger systems

We implemented the NaNet FPGA-based PCIe Gen2 GbE/APElink NIC, featuring GPUDirect RDMA capabilities and UDP protocol management offloading. NaNet is able to receive a UDP input data stream from its GbE interface and redirect it, without any intermediate buffering or CPU intervention, to the memory of a Fermi/Kepler GPU hosted on the same PCIe bus, provided that the two devices share the same upstream root complex. Synthetic benchmarks for latency and bandwidth are presented. We describe how NaNet can be employed in the prototype of the GPU-based RICH low-level trigger processor of the NA62 CERN experiment, to implement the data link between the TEL62 readout boards and the low level trigger processor. Results for the throughput and latency of the integrated system are presented and discussed.

A multi-port 10GbE PCIe NIC featuring UDP offload and GPUDirect capabilities.

NaNet-10 is a four-ports 10GbE PCIe Network Interface Card designed for low-latency real-time operations with GPU systems. To this purpose the design includes an UDP ooad module, for fast and clock-cycle deterministic handling of the transport layer protocol, plus a GPUDirect P2P/RDMA engine for low-latency communication with NVIDIA Tesla GPU devices. A dedicated module (Multi-Stream) can optionally process input UDP streams before data is delivered through PCIe DMA to their destination devices, re-organizing data from dierent streams guaranteeing computational optimization. NaNet-10 is going to be integrated in the NA62 CERN experiment in order to assess the suitability of GPGPU systems as real-time triggers; results and lessons learned while performing this activity will be reported herein.

GPUs for fast triggering and pattern matching at the CERN experiment NA62

In high energy physics experiment the trigger system is crucial to reduce the quantity of data recorded on tape and the acquisition bandwidth requirements. This is particularly true in rare decays experiments. The NA62 experiment aims at measuring the Branching Ratio of K+ -> π+νν, predicted in the Standard Model (SM) at level of ~ 10 -10. In this paper we describe the idea to use the commercial video card processor (GPU) to construct a fast and effective trigger system, both in hardware and software level. Due to the use of off the shelf technology, in continuous development for other purposes, the architecture described would be easily exported to other experiments, to build a versatile and fully customizable trigger system.

Development of Network Interface Cards for TRIDAQ systems with the NaNet framework

NaNet is a framework for the development of FPGA-based PCI Express (PCIe) Network Interface Cards (NICs) with real-time data transport architecture that can be effectively employed in TRIDAQ systems. Key features of the architecture are the flexibility in the configuration of the number and kind of the I/O channels, the hardware offloading of the network protocol stack, the stream processing capability, and the zero-copy CPU and GPU Remote Direct Memory Access (RDMA). Three NIC designs have been developed with the NaNet framework: NaNet-1 and NaNet-10 for the CERN NA62 low level trigger and NaNet3 for the KM3NeT-IT underwater neutrino telescope DAQ system. We will focus our description on the NaNet-10 design, as it is the most complete of the three in terms of capabilities and integrated IPs of the framework.

NaNet: Design of FPGA-based network interface cards for real-time trigger and data acquisition systems in HEP experiments

NaNet is a modular design of a family of FPGA-based PCIe Network Interface Cards specialized for low-latency real-time operations. NaNet features a Network Interface module that implements RDMA-style communications both with the host (CPU) and the GPU accelerators memories (GPUDirect P2P/RDMA) relying on the services of a high performance PCIe Gen3 x8 core. NaNet I/O Interface is highly flexible and is designed for low and predictable communication latency: a dedicated stage manages the network stack protocol in the FPGA logic offloading the host operating system from this task and thus eliminating the associated process jitter effects. Between the two aforementioned modules, stand the data processing and switch modules: the first implements application-dependent processing on streams - e.g. performing compression algorithms - while the second routes data streams between the I/O channels and the Network Interface module. This general architecture has been specialized up to now into three configurations, namely NaNet-1, NaNet3 and NaNet-10 in order to meet the requirements of different experimental setups: NaNet-1 features a GbE channel plus three custom 34 Gbps serial channels and is implemented on the Altera Stratix IV FPGA Development Kit; NaNet3 is implemented on the Terasic DE5-NET Stratix V FPGA development board and supports four custom 2.5 Gbps deterministic latency optical channels; NaNet-10 features four 10GbE SFP+ ports and is also implemented on the Terasic DE5-NET board. We will provide performance results for the three NaNet implementations and describe their usage in the CERN NA62 and KM3NeT-IT underwater neutrino telescope experiments, showing that the architecture is very flexible and yet capable of matching the requirements of low-latency real-time applications with intensive I/O tasks involving the CPU and/or the GPU accelerators.

The TEL62: A real-time board for the NA62 Trigger and Data AcQuisition. Data flow and firmware design

The main goal of the NA62 experiment at CERN SPS is to measure the branching ratio of the ultra-rare K+→π+νν decay, collecting about 100 events in two years of data taking to test the Standard Model of Particle Physics. Readout uniformity of sub-detectors, scalability, efficient online selection and lossless high rate readout are key issues. The TEL62 boards are the common blocks of the NA62 Trigger and Data AcQuisition (TDAQ) system. TEL62s process and store hits coming from the subdetectors in a buffer according to their timestamp, extracting only those requested by the trigger system, which merges trigger primitives also produced by TEL62s. The complete dataflow and firmware organization are described.

Real-time heterogeneous stream processing with NaNet in the NA62 experiment

The use of GPUs to implement general purpose computational tasks, known as GPGPU since fifteen years ago, has reached maturity. Applications take advantage of the parallel architectures of these devices in many different domains. Over the last few years several works have demonstrated the effectiveness of the integration of GPU-based systems in the high level trigger of various HEP experiments. On the other hand, the use of GPUs in the DAQ and low level trigger systems, characterized by stringent real-time constraints, poses several challenges. In order to achieve such a goal we devised NaNet, a FPGA-based PCI-Express Network Interface Card design capable of direct (zero-copy) data transferring with CPU and GPU (GPUDirect) while online processing incoming and outgoing data streams. The board provides as well support for multiple link technologies (1/10/40GbE and custom ones). The validity of our approach has been tested in the context of the NA62 CERN experiment, harvesting the computing power of last generation NVIDIA Pascal GPUs and of the FPGA hosted by NaNet to build in real-time refined physics-related primitives for the RICH detector (i.e. the Cerenkov rings parameters) that enable the building of more stringent conditions for data selection in the low level trigger.

GPU-based low-level trigger system for the standalone reconstruction of the ring-shaped hit patterns in the RICH Cherenkov detector of NA62 experiment

This project aims to exploit the parallel computing power of a commercial Graphics Processing Unit (GPU) to implement fast pattern matching in the Ring Imaging Cherenkov (RICH) detector for the level 0 (L0) trigger of the NA62 experiment. In this approach, the ring-fitting algorithm is seedless, being fed with raw RICH data, with no previous information on the ring position from other detectors. Moreover, since the L0 trigger is provided with a more elaborated information than a simple multiplicity number, it results in a higher selection power. Two methods have been studied in order to reduce the data transfer latency from the readout boards of the detector to the GPU, i.e., the use of a dedicated NIC device driver with very low latency and a direct data transfer protocol from a custom FPGA-based NIC to the GPU. The performance of the system, developed through the FPGA approach, for multi-ring Cherenkov online reconstruction obtained during the NA62 physics runs is presented.