Rudolf Schwarte

New electro-optical mixing and correlating sensor: facilities and applications of the photonic mixer device (PMD)

This paper presents first hardware implementation and investigations of a new electro-optic modulator (EOM), called the Photonic Mixer Device (PMD). The PMD is a semiconductor device combining the characteristics of fast optical sensing and modulation. Arranged to a PMD-matrix it looks like a CCD-matrix but additionally provides the depth information of each image pixel using an appropriately modulated scene illumination. Besides this feature of 3D- imaging by means of time-related correlation the PMD-chip will enable on-chip 2D-image processing by means of spatial correlation. The principle of operation of the PMD, possible technologies for realization, facilities and applications will be described. This new device offers high potential for optical sensory systems due to an amazingly simple and powerful procedure of electro-optical mixing and correlation. Both CCD and CMOS are appropriate technologies. The latter will be treated in detail as a single-element Photonic Mixer Device. Here we propose several architectures--including two quadrant (2Q)-PMDs--with readout and preprocessing circuits for both, the phase resp, time-of-flight values and the pixel intensities. Arranging PMD-pixels to a PMD-line or to a PMD-matrix will provide a new generation of flexible and powerful solid-state 3D- cameras based on time-of-flight. According to the EOM- principle, the proposed 2D-mixer requires no additional optical devices, no broadband electronic amplifiers and mixers. This solid-state PMD-array offers even more unique facilities, e.g., performing high speed spatial light modulation up to the GHz range or optical CDMA- communication. The modulation characteristic of the PMD is an important aspect of this mixer. It is associated with optimizing the layout design for the PMD, which will be presented and discussed in this paper, too. PMD test chips have been realized in CMOS technology. Some simulation and first test results of the chip are also presented. The simulation results prove the operation principle of the PMD and provide us with parameters for an optimized layout design. First experimental results verified the expected operation principle of the test PMD.

New active 3D vision system based on rf-modulation interferometry of incoherent light

Presently there is still a remarkable gap between the requirements and the capabilities of 3D- vision in the field of industrial automation, especially in manufacture integrated 100%-quality control. For these and a lot of other applications like security and traffic control a new extremely fast, precise and flexible 3D-camera concept is presented in this paper. In order to obtain the geometrical 3D information, the whole 3D object or 3D scene is illuminated simultaneously by means of rf-modulated light. This is realized by using optical modulators such as Pockels cells or FTR optical components (FTR: Frustrated Total Reflection). The back scattered light represents the depth information within the local delay of the phase front of the rf-modulated light intensity. If the reflected wave front is mixed again within the whole receiving aperture using the same optical 2D-modulation components and the same rf- frequency, an rf-interference pattern is produced. A CCD camera may be applied to sample these rf-modulation interferograms. In order to reconstruct the 3D-image a minimum of three independent interferograms have to be evaluated. They may be produced either by applying three different rf-phases or three different rf-frequencies. This procedure will be able to deliver up to some tens of high resolution 3D images per second with some hundred thousand voxels (volume elements). Such a remarkable progress can be achieved by means of three key important steps: Firstly by separating the opto-electronic receiver device from real-time requirements by using homodyne mixing of CW-modulated light. Secondly by applying the rf- modulation signal as an optical reference signal to the receiving optical mixer. And thirdly by using a throughout 2D layout of the transmitted illumination, further, of the optical mixer in the receiving aperture, and of the optoelectronic sensing element, e.g., a CCD-chip.

Design of an aspherical lens to generate a homogenous irradiance for three-dimensional sensors with a light-emitting-diode source.

A design of a large-numerical-aperture aspherical singlet for three-dimensional (3-D) sensor applications is presented. This lens can be used to generate a homogenous irradiance on the target in a 3-D sensor, which is based on the principle of time of flight and uses an LED as light source. A numerical method was used in the design. The designed planoaspherical singlet has a numerical aperture of 0.67, low refractive index, and moderate surface shape for easy fabrication. The simulation results revealed that the irradiance deviation within 97% of the designed area is less than 5% and that the transmittance of the lens is greater than 90.5%. The results from a Lambertian source were compared with those from a point source.

New optical four-quadrant phase-detector integrated into a photogate array for small and precise 3D-cameras

The photonic mixer device (PMD) is a new electro-optical mixing semiconductor device. Integrated into a line or an array it may contribute a significant improvement in developing an extremely fast, flexible, robust and low cost 3D-solid-state camera. Three dimensional (3D)-cameras are of dramatically increasing interest in industrial automation, especially for production integrated quality control, in- house navigation, etc. The type of 3D-camera here under consideration is based on the principle of time-of-flight respectively phase delay of surface reflected echoes of rf- modulated light. In contrast to 3D-laser radars there is no scanner required since the whole 3D-scene is illuminated simultaneously using intensity-modulated incoherent light, e.g. in the 10 to 1000 MHz range. The rf-modulated light reflected from the 3D-scene represents the total depth information within the local delay of the back scattered phase front. If this incoming wave front is again rf- modulated by a 2D-mixer within the whole receiving aperture we get a quasi-stationary rf-inference pattern respectively rf-interferogram which may be captured by means of a conventional CCD-camera. This procedure is called rf- modulation interferometry (RFMI). Corresponding to first simulative results the new PMD-array will be appropriate to the RFMI-procedure. Though looking like a modified CCD-array or CMOS-photodetector array it will be able to perform both, the pixelwise mixing process for phase delay respectively depth evaluation and the pixelwise light intensity acquisition for gray level or color evaluation. Further advantageous properties are achieved by means of a four- quadrant (4Q)-PMD array which operates as a balanced inphase/quadrature phase (I/Q)-mixer and will be able to capture the total 3D-scene information of several 100,000 voxels within the microsecond(s) - to ms-range.

New Powerful Sensory Tool in Automotive Safety Systems Based on PMD-Technology

The application of electronic imaging for driver support and occupant safety is actually one of the most challenging tasks in automotive research activities. Most efforts aim to apply two-dimension al imagers in CCD-, CMOS- and TFA-technology so far. Now a further imaging technology has been developed : the PMD-technology, which directly enables 3D-imaging by means of the time-offlight (TOF) principle . This approach offers an extremel y high potential for new solutions in this application field . The PMD-Technology opens up amazing new perspectives for occupant detection systems as well as the observation of vehicles outer area (see Fig . 1.).

3D vision based on PMD-technology for mobile robots

A series of micro-robots (MERLIN: Mobile Experimental Robots for Locomotion and Intelligent Navigation) has been designed and implemented for a broad spectrum of indoor and outdoor tasks on basis of standardized functional modules like sensors, actuators, communication by radio link. The sensors onboard on the MERLIN robot can be divided into two categories: internal sensors for low-level control and for measuring the state of the robot and external sensors for obstacle detection, modeling of the environment and position estimation and navigation of the robot in a global co-ordinate system. The special emphasis of this paper is to describe the capabilities of MERLIN for obstacle detection, targets detection and for distance measurement. Besides ultrasonic sensors a new camera based on PMD-technology is used. This Photonic Mixer Device (PMD) represents a new electro-optic device that provides a smart interface between the world of incoherent optical signals and the world of their electronic signal processing. This PMD-technology directly enables 3D-imaging by means of the time-of-flight (TOF) principle. It offers an extremely high potential for new solutions in the robotics application field. The PMD-Technology opens up amazing new perspectives for obstacle detection systems, target acquisition as well as mapping of unknown environments.

Three-dimensional intelligent sensing based on the PMD technology

With this paper we present new 3D sensing technique based on the novel Photonic Mixer Device (PMD), a new generation of smart 3D sensor, which provides a brilliant interface between the world of incoherent light and the world of electronic signal processing. As a new semiconductor device, it combines fast optical sensing and mixing in one component of pixel size by its unique and powerful principle of operation. Based on standard CMOS-technology, it can be easily integrated into PMD sensing arrays, providing both 3D depth and intensity information of the scene. The presented 3D TOF ranging system based on PMD measures the phase and time delay of the back scattered optical signal. The RF- modulated light reflected from the 3D-scene represents the total 3D depth information within the aperture of the PMD receiver. Since the whole 3D-scene is illuminated simultaneously by using intensity-modulated light, the PMD- array on the receiver side performs parallel electro-optical mixing and correlation and delivers an optimal evaluation of time-of-flight and the optical power for each PMD pixel. So there is no scanner required in contrast to the conventional 3D-laser radar systems. The introduction of the PMD into the 3D range sensing technique offers very attractive solutions for the realization of flexible, extremely fast and robust low-cost 3D solid-state smart ranging systems.

PMD-PLL: receiver structure for incoherent communication and ranging systems

This paper proposes a high integrated optoelectronic phase locked loop for the use in optical data transmission as well as in optical ranging systems. The so called PMD-PLL receiver module is based on a novel electro-optical modulator, called the Photonic Mixer Device (PMD). The sensor is a semiconductor device, which combines fast optical sensing and modulation of incoherent light signals in one component part by its unique and powerful principle of operation. Simulations and experimental results have already verified the operation principle of PMD structures, in CMOS-technology so far. But also other technologies may be suitable for the PMD realization.

MSM-PMD as correlation receiver in a new 3D-ranging system

This contribution describes a ranging system based on a new kind of optical sensor, called the Photonic Mixer Device (PMD). This unit combines fast optical sensing and modulation of incoherent light signals in one component part by its unique and powerful principle of operation. The key feature of this new sensor is the direct detection of the correlation value between the light and an electrical reference signal as an analog voltage value. The realized ranging system is based on the following concept: a laser diode or LED is modulated with a PN-code sequence and the reference signal consists of the same PN-code sequence. With a delay unit the phase of the reference signal is shifted until the PMD-sensor returns an expected correlation value. The delay between the reference and the reflected light signal implies information concerning the distance to the reflecting object. An array of these sensors provides a 3D- ranging system that could be used for, e.g., collision avoidance in new traffic control systems or in product monitoring systems. Fast modulation of the laser and the PMD provides a high distance resolution. Nowadays PMD-realizations are based on standard CMOS technology, but we also keep investigation in a new kind of PMD based on MSM technology (metal- semiconductor-metal).

Electro-optical correlation arrangement for fast 3D cameras: properties and facilities of the electro-optical mixer device

For advanced performances related to 3D metrology in production, security and traffic control, the Institut fur Nachrichtenverarbeitung is currently engaged in the investigation of a new 3D-camera system based on rf- modulation interferometry, other than triangulation. The new 3D-camera system proposed in this paper uses the techniques of optical rf-modulation of the active illumination source and the correlation of these signals reflected from a target within a large aperture. This method yields all phase- correlation functions, relating to each voxel of the 3D- scene in parallel. Hence, the system delivers a fast parallel measurement and evaluation process by completely using a 2D transmitting and receiving path. To fulfill this concept, we have set up a binocular system using a modified Pockels-cell arrangement and a modulated laser transmitter. Since the electro-optical modulator plays a key important role in realizing the whole system, we would discuss the modified arrangement of stacked birefringent KD*P crystals that incorporates the properties of large aperture and high frequency modulation in the region of some tenth of megahertz. Additionally, an optical reference path is introduced to compensate for the fluctuation of the light source, thus enhancing the capability of the system. The experimental results of this reformed system will be presented in this paper, e.g., the standard deviation of (lambda) /2500 being obtained.