Carl L. Fales

Image gathering and processing: information and fidelity

In this paper we formulate and use information and fidelity criteria to assess image gathering and processing, combining optical design with image-forming and edge-detection algorithms. The optical design of the image-gathering system revolves around the relationship among sampling passband, spatial response, and signal-to-noise ratio (SNR). Our formulations of information, fidelity, and optimal (Wiener) restoration account for the insufficient sampling (i.e., aliasing) common in image gathering as well as for the blurring and noise that conventional formulations account for. Performance analyses and simulations for ordinary optical-design constraints and random scenes indicate that (1) different image-forming algorithms prefer different optical designs; (2) informationally optimized designs maximize the robustness of optimal image restorations and lead to the highest-spatial-frequency channel (relative to the sampling passband) for which edge detection is reliable (if the SNR is sufficiently high); and (3) combining the informationally optimized design with a 3 by 3 lateral-inhibitory image-plane-processing algorithm leads to a spatial-response shape that approximates the optimal edge-detection response of (Marrs model of) human vision and thus reduces the data preprocessing and transmission required for machine vision.

Imaging system design for improved information capacity

Shannon’s theory of information for communication channels is used to assess the performance of line-scan and sensor-array imaging systems and to optimize the design trade-offs involving sensitivity, spatial response, and sampling intervals. Formulations and computational evaluations account for spatial responses typical of line-scan and sensor-array mechanisms, lens diffraction and transmittance shading, defocus blur, and square and hexagonal sampling lattices.

Information-theoretic assessment of sampled imaging systems

By rigorously extending modern communication theory to the assessment of sampled imaging systems, we develop the formulations that are required to optimize the performance of these systems within the critical constraints of image gathering, data transmission, and image display. The goal of this optimization is to produce images with the best possible visual quality for the wide range of statistical properties of the radiance field of natural scenes that one normally encounters. Extensive computational results are presented to assess the performance of sampled imaging systems in terms of information rate, theoretical minimum data rate, and fidelity. Comparisons of this assessment with perceptual and measurable performance demonstrate that (1) the information rate that a sampled imaging system conveys from the captured radiance field to the observer is closely correlated with the fidelity, sharpness and clarity with which the observed images can be restored and (2) the associated theoretical minimum data rate is closely correlated with the lowest data rate with which the acquired signal can be encoded for efficient transmission.

An information theory of visual communication

The fundamental problem of visual communication is that of producing the best possible picture at the lowest data rate. We address this problem by extending information theory to the assessment of the visual communication channel as a whole, from image gathering to display. The extension unites two disciplines, the electro- optical design of image gathering and display devices and the digital processing for image coding and restoration. The mathematical development leads to several intuitively attractive figures of merit for assessing the visual communication channel as a function of the critical limiting factors that constrain its performance. Multiresolution decomposition is included in the mathematical development to optimally combine the economical encoding of the transmitted signal with image gathering and restoration. Quantitative and qualitative assessments demonstrate that a visual communication channel ordinarily can be expected to produce the best possible picture at the lowest data rate only if the image-gathering device produces the maximum-realizable information rate and the image-restoration algorithm properly accounts for the critical limiting factors that constrain the visual communication. These assessments encompass (a) the electro-optical design of the image-gathering device in terms of the trade-off between blurring and aliasing in the presence of photodetector and quantization noises, (b) the compression of data transmission by redundancy reduction, (c) the robustness of the image restoration to uncertainties in the statistical properties of the captured radiance field, and (d) the enhancement of particular features or, more generally, of the visual quality of the observed image. The ‘best visual quality’ in this context normally implies a compromise among maximum-realizable fidelity, sharpness, and clarity which depends on the characteristics of the scene and the purpose of the visual communication (e.g. diagnosis versus entertainment).

An information theory of image gathering

Shannons mathematical theory of communication is extended to image gathering. The performance of image gathering, as well as of natural vision, is critically constrained by the realizability of the spatial-frequency response of optical apertures, the sampling passband of the photon-detection mechanism, and the signal-to-noise ratio. Hence, whereas Shannon could assume sufficient sampling, we have to deal with insufficient sampling. This extension requires a careful examination of the basic principles of information theory to obtain an unambiguous quantitative description of information for undersampled systems. In this paper we obtain expressions for the total information that is received with a single image-gathering channel and with parallel channels. Some of these expressions challenge intuition, but, when properly interpreted, have clear and precise meaning. The thrust of this meaning leads inexorably to the conclusion that the aliased signal components carry information even though these components interfere with the within-passband signal components in conventional image gathering and restoration, thereby degrading the fidelity and visual quality of the restored image. But a close examination of the expression for minimum mean-square-error, or Wiener-matrix, restoration from parallel image-gathering channels also reveals a method for unscrambling the within-passband and aliased signal components to restore spatial frequencies beyond the sampling passband out to the spatial-frequency response cutoff of the optical aperture. This method requires us to gather K discrete images, each with a different image-gathering response, i.e., channel as obtained, for example, by changing the objective lens aperture, to restore images with a resolution that is √K times finer than the sampling interval.

Image Gathering and Digital Restoration

This paper seeks to unite two disciplines: the electro-optical design of image gathering and display devices and the digital processing for image restoration. So far, these two disciplines have remained independent, following distinctly separate traditions. However, the best possible performance can be attained only when the digital processing algorithm accounts for the critical limiting factors of image gathering and display and the image-gathering device is designed to enhance the performance of the digital-processing algorithm. The following salient advantages accrue: 1. Spatial detail as fine as the sampling interval of the image-gathering device ordinarily can be restored sharply and clearly. 2. Even finer spatial detail than the sampling interval can be restored by combining a multiresponse image-gathering sequence with a restoration filter that properly reassembles the within-passband and aliased signal components. 3. The visual quality produced by traditional image gathering (e.g. television camera) and reconstruction (e.g. cubic convolution) can be improved with a small-kernel restoration operator without an increase in digital processing. 4. The enhancement of radiance-field transitions can be improved for dynamicrange compression (to suppress shadow obscurations) and for edge detection (for computer vision).

Visual communication with retinex coding.

Visual communication with retinex coding seeks to suppress the spatial variation of the irradiance (e.g., shadows) across natural scenes and preserve only the spatial detail and the reflectance (or the lightness) of the surface itself. The separation of reflectance from irradiance begins with nonlinear retinex coding that sharply and clearly enhances edges and preserves their contrast, and it ends with a Wiener filter that restores images from this edge and contrast information. An approximate small-signal model of image gathering with retinex coding is found to consist of the familiar difference-of-Gaussian bandpass filter and a locally adaptive automatic-gain control. A linear representation of this model is used to develop expressions within the small-signal constraint for the information rate and the theoretical minimum data rate of the retinex-coded signal and for the maximum-realizable fidelity of the images restored from this signal. Extensive computations and simulations demonstrate that predictions based on these figures of merit correlate closely with perceptual and measured performance. Hence these predictions can serve as a general guide for the design of visual communication channels that produce images with a visual quality that consistently approaches the best possible sharpness, clarity, and reflectance constancy, even for nonuniform irradiances. The suppression of shadows in the restored image is found to be constrained inherently more by the sharpness of their penumbra than by their depth.

Image-plane processing of visual information.

Shannon’s theory of information is used to optimize the optical design of sensor-array imaging systems which use neighborhood image-plane signal processing, similar to the lateral inhibitory preprocessing in natural vision, for enhancing edges and compressing dynamic range during image formation. The resultant edge-enhancement, or bandpass-filter, response is found to be similar to that of (Marr’s model of) human vision. Comparisons of traits in natural vision with results from information theory suggest that image-plane processing can improve visual information acquisition for pattern recognition when resolving power, sensitivity, and dynamic range are constrained. The improvements that can be attained for constructing edge-enhanced images and primal sketches include reduced sensitivity to changes in light levels reduced data transmission, and reduced data processing.

Visual Communication: Information and Fidelity

This assessment of visual communication deals with image gathering, coding, and restoration as a whole rather than as separate and independent tasks. The approach focuses on two mathematical criteria, information and fidelity, and on their relationships to the entropy of the encoded data and to the visual quality of the restored image. Past applications of these criteria to the assessment of image coding and restoration have been limited to the link that connects the output of the image-gathering device to the input of the image-display device. By contrast, the approach presented in this paper explicitly includes the critical limiting factors that constrain image gathering and display. This extension leads to an end-to-end assessment theory of visual communication that combines optical design with digital processing.

Electro-optical design for efficient visual communication

Friedrich O. Huck, MEMBER SPIECarl L. FalesDaniel J. JobsonNASA Langley Research CenterHampton, Virginia 23681E-mail: f.o.huck @larc.nasa.govZia-ur RahmanScience and Technology CorporationHampton, Virginia 23666Abstract. Visual communication, in the form of telephotography andtelevision, for example, can be regarded as efficient only if the amountof information that it conveys about the scene to the observer ap-proaches the maximum possible and the associated cost approachesthe minimum possible. Elsewhere we have addressed the problem ofassessing the end-to-end performance of visual communication systemsin terms of their efficiency in this sense by integrating the critical limitingfactors that constrain image gathering into classical communication the-ory. We use this approach to assess the electro-optical design of image-gathering devices as a function of the f number and apodization of theobjective lens and the aperture size and sampling geometry of the pho-todetection mechanism. Results show that an image-gathering devicethat is designed to optimize information capacity performs similarly to thehuman eye. For both, the performance approaches the maximum pos-sible, in terms of the efficiency with which the acquired information canbe transmitted as decorrelated data, and the fidelity, sharpness, andclarity with which fine detail can be restored.Subject terms: electro-optical design: information; entropy; dynamic-rangecompression; image coding; image restoration.Optical Engineering 34(3), 795-813 (March 1995).1 IntroductionThe problem of visual communication is that of producingan image that conveys intormation to the human observer atone point about a scene that is located at another point. Untilrecently, in telephotography and television, fiw example, theinput terminal of the visual communication channel consistedsolely of the image-gathering device that transtk_rms the spa-lially varying radiance field reflected or emitted by the sceneinto the signal that is transmitted, and the output terminalconsisted solely of the image display device that transformsthe received signal into an image. However. advances intechnology are leading to rapid growth in the capabilities ofanalog and digital VLSI processors, even as their cost, size,weight, and power consumption decrease. Consequently, vis-ual communication is now increasingly carried out by com-bining image gathering and display with digital image pro-cessing, hnage gathering is combined with encoding toreduce data transmission, and image display is combined withrestoration to enhance image quality. So far, however, theelectro-optical design of image-gathering devices and thedigital image processing for encoding and restoration haveremained independent disciplines, following distinctly sep-arate traditions.The electro-optical design of image-gathering devices or-dinarily revolves around two interdependent trade-offs. Onetrade-off, in terms of geometrical optics, is widely under-stood. It deals with instantaneous lield of view (IFOV) versus