Michele Caselle

A PCIe DMA Architecture for Multi-Gigabyte Per Second Data Transmission

We developed a direct memory access (DMA) engine compatible with the Xilinx PCI Express (PCIe) core to provide a high-performance and low-occupancy alternative to commercial solutions. In order to maximize the PCIe throughput while minimizing the FPGA resources utilization, the DMA engine adopts a novel strategy where the DMA address list is stored inside the FPGA and not in the central memory of the host CPU. The FPGA design package is complemented with simple register access to control the DMA engine by a Linux driver. The design is compatible with Xilinx FPGA Families 6 and 7, and operates with the Xilinx PCIe endpoint Generation 1 and 2 with all lane configurations (x1, x2, x4, x8). A multi-engine architecture is also presented, where two x8 lanes cores are used in parallel together with a PCIe bridge, to exploit fully the capabilities of a PCIe Gen2 x16 lanes link. A data throughput of 3461 MBytes/s has been achieved with a single PCIe Gen2 x8 lanes endpoint. If the dual-engine architecture is used, the throughput is increased up to 6920 MBytes/s. The presented DMA is currently used in several experiments at the ANKA synchrotron light source.

An ultra-fast data acquisition system for coherent synchrotron radiation with terahertz detectors

The recording of coherent synchrotron radiation requires data acquisition systems with a temporal resolution of tens of picosecond. This paper describes a new real-time and high-accuracy data acquisition system suitable for recording individual ultra-short pulses generated by a fast terahertz (THz) detector (e.g. YBCO, NbN, Zero Biased Schottky Diode). The system consists of a fast sampling board combined with a high data throughput readout. The first board is designed for sampling the fast pulse signals with a full width half maximum (FWHM) between a few tens to one hundred picoseconds with a minimum sampling time of 3 ps. The high data throughput board consists of a PCIe-Bus Master DMA architecture used for fast data transfer up to 3 GByte/s. The full readout chain with fast THz detectors and the acquisition system has been successfully tested at the synchrotron ANKA. An overview of the electronics system and preliminary results with multi-bunch filling pattern will be presented.

A new DMA PCIe architecture for Gigabyte data transmission

PCI Express (PCIe) is a high-speed serial point-to-point interconnect that delivers high-performance data throughput. KIT has developed a Direct Memory Access (DMA) engine compatible with the Xilinx PCIe core to provide a smart and low-occupancy alternative logic to expensive commercial solutions. In order to maximize the PCIe throughput the DMA engine adopts a new strategy, where the DMA descriptor list is stored inside the FPGA and not in the central memory system. The FPGA design package is complemented with a simple register access to control the DMA engine by a Linux driver. A handshaking sequence between the DMA engine and the Linux driver ensures that no errors occure, even in data transfers of several hundreds of Gigabytes. The design has been tested with Xilinx FPGA Families 6 and 7, and operates with the Xilinx PCIe endpoint generation 1 and 2 with all lane configurations (x1, x2, x4, x8, x16). Data throughput of more than 3.4 GB/s has been achieved with a PCIe Gen 2 ×8 lanes endpoint. The proposed DMA is currently used in several experiments at the ANKA synchrotron light source.

A high-throughput readout architecture based on PCI-Express Gen3 and DirectGMA technology

Modern physics experiments produce multi-GB/s data rates. Fast data links and high performance computing stages are required for continuous data acquisition and processing. Because of their intrinsic parallelism and computational power, GPUs emerged as an ideal solution to process this data in high performance computing applications. In this paper we present a high-throughput platform based on direct FPGA-GPU communication. The architecture consists of a Direct Memory Access (DMA) engine compatible with the Xilinx PCI-Express core, a Linux driver for register access, and high- level software to manage direct memory transfers using AMDs DirectGMA technology. Measurements with a Gen3 x8 link show a throughput of 6.4 GB/s for transfers to GPU memory and 6.6 GB/s to system memory. We also assess the possibility of using the architecture in low latency systems: preliminary measurements show a round-trip latency as low as 1 μs for data transfers to system memory, while the additional latency introduced by OpenCL scheduling is the current limitation for GPU based systems. Our implementation is suitable for real-time DAQ system applications ranging from photon science and medical imaging to High Energy Physics (HEP) systems.

Fast Mapping of Terahertz Bursting Thresholds and Characteristics at Synchrotron Light Sources

Dedicated optics with extremely short electron bunches enable synchrotron light sources to generate intense coherent THz radiation. The high degree of spatial compression in this so-called low-αc optics entails a complex longitudinal dynamics of the electron bunches, which can be probed studying the fluctuations in the emitted terahertz radiation caused by the micro-bunching instability (“bursting”). This article presents a “quasi-instantaneous” method for measuring the bursting characteristics by simultaneously collecting and evaluating the information from all bunches in a multi-bunch fill, reducing the measurement time from hours to seconds. This speed-up allows systematic studies of the bursting characteristics for various accelerator settings within a single fill of the machine, enabling a comprehensive comparison of the measured bursting thresholds with theoretical predictions by the bunched-beam theory. This paper introduces the method and presents first results obtained at the ANKA synchrotron radiation facility.

A Control System and Streaming DAQ Platform with Image-Based Trigger for X-ray Imaging

High-speed X-ray imaging applications play a crucial role for non-destructive investigations of the dynamics in material science and biology. On-line data analysis is necessary for quality assurance and data-driven feedback, leading to a more efficient use of a beam time and increased data quality. In this article we present a smart camera platform with embedded Field Programmable Gate Array (FPGA) processing that is able to stream and process data continuously in real-time. The setup consists of a Complementary Metal-Oxide-Semiconductor (CMOS) sensor, an FPGA readout card, and a readout computer. It is seamlessly integrated in a new custom experiment control system called Concert that provides a more efficient way of operating a beamline by integrating device control, experiment process control, and data analysis. The potential of the embedded processing is demonstrated by implementing an image-based trigger. It records the temporal evolution of physical events with increased speed while maintaining the full field of view. The complete data acquisition system, with Concert and the smart camera platform was successfully integrated and used for fast X-ray imaging experiments at KITs synchrotron radiation facility ANKA.

KAPTURE-2. A picosecond sampling system for individual THz pulses with high repetition rate

This paper presents a novel data acquisition system for continuous sampling of ultra-short pulses generated by terahertz (THz) detectors. Karlsruhe Pulse Taking Ultra-fast Readout Electronics (KAPTURE) is able to digitize pulse shapes with a sampling time down to 3 ps and pulse repetition rates up to 500 MHz. KAPTURE has been integrated as a permanent diagnostic device at ANKA and is used for investigating the emitted coherent synchrotron radiation in the THz range. A second version of KAPTURE has been developed to improve the performance and flexibility. The new version offers a better sampling accuracy for a pulse repetition rate up to 2 GHz. The higher data rate produced by the sampling system is processed in real-time by a heterogeneous FPGA and GPU architecture operating up to 6.5 GB/s continuously. Results in accelerator physics will be reported and the new design of KAPTURE be discussed.

Low-cost bump-bonding processes for high energy physics pixel detectors

In the next generation of collider experiments detectors will be challenged by unprecedented particle fluxes. Thus large detector arrays of highly pixelated detectors with minimal dead area will be required at reasonable costs. Bump-bonding of pixel detectors has been shown to be a major cost-driver. KIT is one of five production centers of the CMS barrel pixel detector for the Phase I Upgrade. In this contribution the SnPb bump-bonding process and the production yield is reported. In parallel to the production of the new CMS pixel detector, several alternatives to the expensive photolithography electroplating/electroless metal deposition technologies are developing. Recent progress and challenges faced in the development of bump-bonding technology based on gold-stud bonding by thin (15 μm) gold wire is presented. This technique allows producing metal bumps with diameters down to 30 μm without using photolithography processes, which are typically required to provide suitable under bump metallization. The short setup time for the bumping process makes gold-stud bump-bonding highly attractive (and affordable) for the flip-chipping of single prototype ICs, which is the main limitation of the current photolithography processes.

Novel module production methods for the CMS pixel detector, upgrade phase I

For the High-Luminosity upgrade of the LHC (HL-LHC), phase I, the CMS pixel detector needs to be replaced. In order to improve the tracking resolution even at high luminosity the pixel detector is upgraded by a fourth barrel layer. This paper describes the production process and results for the fourth barrel layer for the CMS silicon pixel detector, upgrade phase I. The additional barrel layer will be produced by KIT and DESY. Both research centers have commonly developed and investigated new production processes, including SAC solder bump jetting, gold stud bumping and Precoat by Powder Processes (PPS) to bump the sensor tiles and prepare them for the flip-chip process. First bare modules have been produced with the new digital ROC.

An FPGA-based track finder for the L1 trigger of the CMS experiment at the HL-LHC

A new tracking detector is under development for use by the CMS experiment at the High- Luminosity LHC (HL-LHC). A crucial component of this upgrade will be the ability to reconstruct within a few microseconds all charged particle tracks with transverse momentum above 3 GeV, so they can be used in the Level-1 trigger decision. A concept for an FPGA-based track finder using a fully time-multiplexed architecture is presented, where track candidates are reconstructed using a projective binning algorithm based on the Hough Transform followed by a track fitting based on the linear regression technique. A hardware demonstrator using MP7 processing boards has been assembled to prove the entire system, from the output of the tracker readout boards to the reconstruction of tracks with fitted helix parameters. It successfully operates on one eighth of the tracker solid angle at a time, processing events taken at 40 MHz, each with up to 200 superimposed proton-proton interactions, whilst satisfying latency constraints. The demonstrated track-reconstruction system, the chosen architecture, the achievements to date and future options for such a system will be discussed.