Takeharu Goji Etoh

Toward One Giga Frames per Second — Evolution of in Situ Storage Image Sensors

The ISIS is an ultra-fast image sensor with in-pixel storage. The evolution of the ISIS in the past and in the near future is reviewed and forecasted. To cover the storage area with a light shield, the conventional frontside illuminated ISIS has a limited fill factor. To achieve higher sensitivity, a BSI ISIS was developed. To avoid direct intrusion of light and migration of signal electrons to the storage area on the frontside, a cross-sectional sensor structure with thick pnpn layers was developed, and named “Tetratified structure”. By folding and looping in-pixel storage CCDs, an image signal accumulation sensor, ISAS, is proposed. The ISAS has a new function, the in-pixel signal accumulation, in addition to the ultra-high-speed imaging. To achieve much higher frame rate, a multi-collection-gate (MCG) BSI image sensor architecture is proposed. The photoreceptive area forms a honeycomb-like shape. Performance of a hexagonal CCD-type MCG BSI sensor is examined by simulations. The highest frame rate is theoretically more than 1Gfps. For the near future, a stacked hybrid CCD/CMOS MCG image sensor seems most promising. The associated problems are discussed. A fine TSV process is the key technology to realize the structure.

Bubble entrapment through topological change

When a viscous drop impacts onto a solid surface, it entraps a myriad of microbubbles at the interface between liquid and solid. We present direct high-speed video observations of this entrapment. For viscous drops, the tip of the spreading lamella is separated from the surface and levitated on a cushion of air. We show that the primary mechanism for the bubble entrapment is contact between this precursor sheet of liquid with the solid and not air pulled directly through cusps in the contact line. The sheet makes contact with the solid surface, forming a wetted patch, which grows in size, but only entraps a bubble when it meets the advancing contact line. The leading front of this wet patch can also lead to the localized thinning and puncturing of the liquid film producing strong splashing of droplets.

A 16 Mfps 165kpixel backside-illuminated CCD

In 2002, we reported a CCD image sensor with 260×312 pixels capable of capturing 103 consecutive images at 1,000,000 frames per second (1Mfps) [1]. We named the sensor “ISIS-V2”, for In-situStorage Image Sensor Version 2. 103 memory elements are attached to every pixel; generated image signals were instantly and continuously stored in the in-situstorage without being read out of the sensor. The ultimate high-speed recording was enabled by this parallel recording at all pixels. In 2006, the color version, ISIS-V4, was reported [2]. In 2009, we developed ISIS-V12, a backside-illuminated image sensor mounting the ISIS structure and the CCM, charge-carrier multiplication, on the front side [3]. The CCM is a CCD-specific efficient signal-amplification device. CCM, combined with the BSI structure and cooling, achieved very high sensitivity. The ISIS-V12 was a test sensor intended to prove the technical feasibility of the structure. The maximum frame rate was 250kfps for a charge-handling capacity of Qmax=10,000e− and 1Mfps for a reduced Qmax. The pixel count was 489×400 pixels. For backside-illuminated (BSI) image sensors, metal wires can be placed on the front surface to increase the frame rate without reducing fill factor or violating uniformity of the pixel configuration. It has been proved by simulations that 100Mfps is achievable by introducing innovative technologies including a special wiring method [4]. We now report on ISIS-V16, developed by incorporating technologies to increase the frame rate with those to achieve very high sensitivity, which was confirmed by evaluation of ISIS-V12. The performance specification of ISIS-V16 is summarized in Fig. 23.4.1.

A 252-

We developed a 312-kpixel back-side-illuminated ultrahigh-speed charge-coupled device (CCD) that has a sensitivity of 252 V/lux · s and is capable of operating at 16.7 Mfps. The potential profile of the pixel was designed by using a 3-D semiconductor device simulator. The high sensitivity results from the unit having fill factor and time aperture ratios of 100% and a high optical utilization ratio. Its sensitivity is 12.7 times that of a front-side-illuminated image sensor. Ultrahigh-speed shooting was enabled by an in situ storage image sensor. By reducing the wiring resistance and dividing the image area into eight blocks, a maximum frame rate of 16.7 Mfps was attained. The total pixel count is 760 horizontally and 411 vertically. The burst capturing speed is thus 5.2 Tpixel/s, making it the fastest imaging device to date.

{\rm V/lux}{\cdot}{\rm s}

The frame rate of the digital high-speed video camera was 2000 frames per second (fps) in 1989, and has been exponentially increasing. A simulation study showed that a silicon image sensor made with a 130 nm process technology can achieve about 1010 fps. The frame rate seems to approach the upper bound. Rayleigh proposed an expression on the theoretical spatial resolution limit when the resolution of lenses approached the limit. In this paper, the temporal resolution limit of silicon image sensors was theoretically analyzed. It is revealed that the limit is mainly governed by mixing of charges with different travel times caused by the distribution of penetration depth of light. The derived expression of the limit is extremely simple, yet accurate. For example, the limit for green light of 550 nm incident to silicon image sensors at 300 K is 11.1 picoseconds. Therefore, the theoretical highest frame rate is 90.1 Gfps (about 1011 fps).

, 16.7-Million-Frames-Per-Second 312-kpixel Back-Side-Illuminated Ultrahigh-Speed Charge-Coupled Device

An image sensor for an ultra-high-speed video camera was developed. The maximum frame rate, the pixel count and the number of consecutive frames are 1,000,000 fps, 720 x 410 (= 295,200) pixels, and 144 frames. A micro lens array will be attached on the chip, which increases the fill factor to about 50%. In addition to the ultra-high-speed image capturing operation to store image signals in the in-situ storage area adjacent to each pixel, standard parallel readout operation at 1,000 fps for full frame readout is also introduced with sixteen readout taps, for which the image signals are transferred to and stored in a storage device with a large capacity equipped outside the sensor. The aspect ratio of the frame is about 16 : 9, which is equal to that of the HDTV format. Therefore, a video camera with four sensors of the ISIS-V4, which are arranged to form the Bayer’s color filter array, realizes an ultra-high-speed video camera of a semi-HDTV format.

The Theoretical Highest Frame Rate of Silicon Image Sensors

In this paper, a 3D 1Gfps BSI image sensor is proposed, where 128 × 256 pixels are located in the top-tier chip and a 32 × 32 localized driver array in the bottom-tier chip. Pixels are designed with Multiple Collection Gates (MCG), which collects photons selectively with different collection gates being active at intervals of 1ns to achieve 1Gfps. For the drivers, a global PLL is designed, which consists of a ring oscillator with 6-stage current starved differential inverters, achieving a wide frequency tuning range from 40MHz to 360MHz (20ps rms jitter). The drivers are the replicas of the ring oscillator that operates within a PLL. Together with level shifters and XNOR gates, continuous 3.3V pulses are generated with desired pulse width, which is 1/12 of the PLL clock period. The driver array is activated by a START signal, which propagates through a highly balanced clock tree, to activate all the pixels at the same time with virtually negligible skew.

An image sensor of 1,000,000 fps, 300,000 pixels, and 144 consecutive frames

Crosstalk in the backside-illuminated multi-collection-gate (BSI-MCG) image sensor was analyzed by means of Monte Carlo simulation. The BSI-MCG image sensor was proposed to achieve the temporal resolution of 1 ns. In this sensor, signal electrons generated by incident light near the back side travel to the central area of the pixel on the front side. Most of the signal electrons are collected by a collecting gate, to which a higher voltage is applied than that of other collection gates. However, due to spatial and temporal diffusion, some of the signal electrons migrate to other collection gates than the collecting gate, resulting in spatiotemporal crosstalk, i.e., mixture of signal electrons at neighboring collection gates and/or pixels. To reduce the crosstalk, the BSI-MCG structure is modified and the performance is preliminarily evaluated by Monte Carlo simulation. An additional donut-shaped N type implantation at the collection-gate area improves the potential gradient to the collecting gate, which reduces the crosstalk caused by the spatial diffusion. A multi-framing camera based on the BSI-MCG image sensor can be applied to Fluorescence Lifetime Imaging Microscopy (FLIM). In this case, crosstalk reduces accuracy in estimation of the lifetimes of fluorophore samples. The inaccuracy is compensated in a post image processing based on a proposed impulse response method.

Pixel parallel localized driver design for a 128 x 256 pixel array 3D 1Gfps image sensor

The first digital high-speed video camera was developed in 1989. Since then, the highest frame rate has exponentially increased from about 103 fps (frames per second) to 108 fps, and will reach 109–1010 fps in the very near future. The evolution has been supported by successive innovations of the technology: (1) in the middle of the 1980s, solid-state image sensors enabling parallel and partial readout were introduced, (2) in 1989, a continuous-readout digital-recording high-speed video camera was developed, (3) 1996, a burst image sensor with in-pixel SPS-CCD storage was invented, (4) in 2002, a burst image sensor with in-pixel slanted linear CCD storage achieved 1 × 106 fps (1 Mfps), (5) in 2011, a backside-illuminated burst image sensor significantly increased the sensitivity and also the frame rate to 16 Mfps, (6) in 2012, a CMOS burst image sensor with the pixel-based storage in the periphery of the chip was developed, (7) in 2015, a macro-pixel multi-framing image sensor achieved a frame interval of 5 ns (200 Mfps), and, (8) currently, the 3D-stacking technology is further increasing the frame rates. The history is reviewed with descriptions of the epoch-making achievements in each stage and works going on. When the spatial resolution of lenses approached the limit, Rayleigh discussed the theoretical spatial resolution limit. It’s time to search for the temporal resolution limit of high-speed image sensors. The limit for the silicon image sensors was theoretically derived. For example, the temporal resolution limit for green light of 550 nm is 11.1 ps. Therefore, the theoretical highest frame rate is about 1011 fps.

Crosstalk in multi-collection-gate image sensors and its improvement

Particle Tracking Velocimetry, PTV, is one of the most powerful tools those can measure the deformation of fast flowing fluid, such as vorticity, shear rate, and so on, with a high-speed video camera. We have developed a new method to estimate vorticity from randomly located velocity vectors obtained by a PTV. The proposed method employs the Moving Least Square method that is developed for the meshless numerical simulation. The optimal size of fitting area of the proposed method is derived theoretically and is confirmed by the Monte Carlo simulation. The comparison of accuracy of the proposed method is carried out. The result shows that the proposed method is more accurate than the commonly used method in any case.