Image Processing

Type of physical science: Mathematical methods

Field of study: Signal processing

Image processing (computer vision) provides digital signal processing of two- or higher-dimensional signals. Image processing can range from simple filtering enhancement of an individual image to more complicated image-data comparisons and pattern recognition. Image analysis is the converse of computer graphics, since the input is a model of a scene as it would be seen from a given viewing location, and its output a true scene description.

Overview

Image processing or analysis is a multidisciplinary field with methods and jargon often overlapping with signal processing. An image in this context is a two-dimensional representation of a measured or observed object or quantity, obtained by means such as photography, X-ray radiography and tomography, ultrasound, or radar. Although different for different applications, image processing entails a sequence of generic computer-processing operations: data digitization and compression, image enhancement and reconstruction, and image segmentation, matching, and pattern recognition.

In most cases, image processing converts measured density levels on a sensor or film to numbers enterable into computer memory. The number of picture elements, or pixels, needed to represent a given image accurately depends on overall image size, the number and kind of fine details, and structural complexity. This input digitization comprises both sampling and quantization. In sampling, the two-dimensional grid spacing must be sufficiently fine to pick up all important image details. Employing too few discrete sample points can give rise to poorly separated or smeared image features.

The Nyquist sampling theorem requires a grid sampling size of d to reconstruct all image components of period 2d. In many images, point sampling (quantization) can be coarse in regions with (slow) gray-level variation; these facts can be used in an adaptive sampling/quantization algorithm which varies the number and size of data samples from image point to point. This sampling generates an integer for each pixel representing intensity or amplitude at that position; some multispectral images from satellites, for example, can have more than one local signal property defined at each point. Quantization represents the main image variable (height, brightness, and so on) at a given (x,y) point by a given z value. When sampling and quantization are completed for all pixels, the image is digitally represented by a rectangular array of numbers, which can subsequently be processed using digital signal- and related image-processing techniques.

The choice of specific image-analysis techniques in any given case depends strongly on the data quality and particular application. Practical solution of many image-processing problems frequently results from using a combination of techniques in a proper sequence. Most image-processing techniques seek to reduce complicated object shapes to as few descriptive parameters as possible. Topographic roughness as measured by radar or satellite mapping, for example, is frequently specified by storing values of feature location and shape, local root-mean-square height, autocorrelation length, and Fourier transform power spectral coefficients.

Aspects of the more general inverse problem often arise in image processing, since there are usually many possible objects that could give rise to a given image. In most images, both shape and intensity properties are considered. Early in the image-processing sequence, simple statistical measures, such as histograms of the digitized intensity distribution within an image, are invaluable for performing image intensity and contrast comparisons for optimal display.

Frequently, the discrete fast Fourier transform is employed to decompose a complex image pattern quantitatively into a sum of simple shapes of different spatial frequency. Here, large amplitudes at high spatial frequencies indicate highly curved boundaries or abrupt edges; low-frequency components are interpreted as overall gradual trends. Filters in general suppress much of the random image noise and fluctuations, as well as clarify image details and reduce some of the computational noise of subsequent digital image-processing sequences. High (low) pass filtering removes slowly (rapidly) varying components and retains only high-frequency local (low-frequency global) variations. As in signal processing, high and low pass filtering is usually performed computationally by a local moving average statistic.

Image enhancement is vital because object illumination is usually too high or too low, and thus gray-level-inoptimum, to show the full image against the background. Image enhancement is implemented in either the time or frequency (transform) domain. In the time domain, smoothing and image-sharpening operators are frequently used. One basic image enhancer is thus gray scale modification, accomplished by spreading the image gray levels over a wider range. Another, related technique is to map background gray levels into a greater number of colors.

In the transform domain, low pass and homomorphic filtering are typically used. Since image illumination (low frequency) and reflectivity (high frequency) vary slowly and rapidly respectively, in homomorphic filtering their independent variations can be separated by taking the natural logarithm of each point's gray level. Then the fast Fourier transform of the log-scaled image is taken, and high-frequency enhancing filters are applied, where the result is antilogged and inverse Fourier transformed. The general goal in all image restoration is removing or undoing unwanted effects in the image that result from geometric and/or noise degradation. More advanced image restoration frequently employs inversion techniques, as well as Kalman filtering, constrained optimum filtering, and recursive (Wiener) filtering. Least square Wiener inverse filtering uses the standard convolutional model g = h x f + n, where f is the desired true image, h the measuring system or medium response, n statistically governed noise, and g the actually measured image. Kalman filtering estimates each image pixel value as a linear combination of estimates at nearby pixels, such that weighting coefficients have minimum squared error.

Image reconstruction uses various so-called projection techniques to sum gray/intensity levels along specific groups or sections to form a more correct total image than the previous image estimates. The one-dimensional Fourier transform of a projection of an image is the cross section of a two-dimensional image transform. The full Fourier transform can be approximated by interpolating from available cross sections and finally reconstructing the image by inverse Fourier transformation. Another well-known image-reconstruction technique is that of back-projection. Here, each image projection gives a set of linear algebraic equations in the gray-level variable, which when solved yield a higher-quality total image estimate.

Image segmentation relies on region and edge detection methods. Region detection is accomplished via simple and more complex "thresholding" methods. Thresh-old methods define a min/max local gray-scale level, or rate of change of gray level per unit distance, which, when exceeded, is taken to define a new or different image segment. Edges are detected by so-called local contrast gradients, using statistical models of what expected types of edges should look like in terms of gray level. Contrast gradients permit edge definition by computing the second spatial derivative of gray-level values.

Image recovery seeks an even more highly refined image by separating from the net recorded image the effects on gray scale of object illumination, object reflectivity, and object orientation. In this stage, true image shape can be determined from ray-theoretic shadows and shading, image texture, and model objects and shapes. Gray-level direction gradients resulting from object and surface roughness are used to infer object roughness features.

Pattern recognition is usually the final stage of image processing. By "pattern" is meant some structure or form present in a given signal data set. Examples of patterns in science and engineering include the waveshape of a seismic or sonar signal in contact with a particular reflecting interface. Pattern recognition in general can be divided into decision-theoretic and structural methods. Decision-theoretic pattern recognition is mainly concerned with classification of patterns, via algorithmically assigning an observed signal or image pattern to an already established class (or classes) to which it is most likely to belong. If the patterns in question have a more complex structure, such as contour lines or geometric surfaces, then, in addition to pattern classification, pattern analysis is needed. Here the pattern is decomposed into sub-patterns, sub-patterns into sub-subpatterns, and so on, until a stable set of elementary patterns is achieved.

Pattern recognition within images frequently employs Fourier transform analysis, matched filtering, and, more recently, (multi)fractal methods. Fourier series and fast Fourier transform (FFT) analysis permit decomposing and selectively reconstructing complex images as a function of spatial wavelength and angular orientation. Matched filtering is a technique based on least-squares matching of a mathematical function to a given image component. Fractals are mathematical models of one-, two-, or three-dimensional curves, shapes, and other physical boundaries, which assume that a given geometric pattern does not change with positive or negative magnification (scale-invariant).

Many simpler patterns can be detected by comparing the observed image with a prior template or model using the cross-correlation function, by subtractive differencing, stereo-imaging, or triangulation. In pattern recognition, an object is taken to be a structure or arrangement of parts whose relations or properties satisfy different geometrical, topological, and other constraints. Pattern and feature detection typically begin seeking spatial pattern types in the observed image with a collection of templates based on computing first/second spatial derivatives, autocorrelation, or spectral functions. Image segmentation further identifies other distinct (sub)regions and properties of pixels within and connecting the above basic image units.

The groups of thus-labeled patterns and segments can then be regrouped and partitioned within and connecting the above basic image sub-patterns.

In so-called syntactic pattern recognition, patterns are identified by detecting various elements or basic image components satisfying a number of constraints, based on a priori knowledge of what a given class of objects is expected to look like.

Applications

The practical possibilities and utility of many image-processing operations depends on the ease or difficulty with which digital computers can perform binary operations in general, and Fourier transform type computations in particular.

Examples of applied image processing and analysis include medical imaging, such as analysis of X-ray photography, scintigrams, ultrasonic and NMR (nuclear magnetic resonance) measurements, as well at CAT-scans; related (forensic) applications include fingerprint, voice, and profile analysis. In the realm of classical and quantum physics, uses of digital image processing include electron microscopy and various types of optical and accoustical interferometry. The basic products of meteorology, contoured maps of surface temperature, humidity, and barometric pressure, are also heavily dependent upon image processing and display. Many geophysical image applications--such as gravity and magnetic contour maps, seismic and acoustic wavefields, and remote-sensing satellite-telemetered visual, radar, and microwave band images--exploit methods to translate two-dimensional images into a series of parallel one-dimensional signals.

Tomography, another developing area of digital image processing, is the reconstruction or imaging of a three-dimensional object or volume by using measurements of waves passing through the object. In X-ray and seismic tomography, for example, the estimates of tissue density and earth layer velocities are reconstructed from measurements of X-ray absorption and seismic travel time. In several high-resolution forms of scanning microscopy, images of surfaces are obtained by applying a special scanning signal to x- and y-dimension transducers; the resulting image in x, y, and z space is a combination of material geometry and atomic structure. More generally, digital imaging microscopy, widely used in almost all the above applications and representative of many imaging approaches, is notably improved by permitting routine detection of very low light levels and of a quantized image intensity.

Background noise, sensor nonlinearity, and other distortions are algorithmically removed before focusing, and both contrast stretching and thresholding are subsequently employed to isolate and emphasize particular image features.

Despite the common use of the gray scale or color to measure and define image intensity, the features that most images map are not primarily shadow, color, or topographic surface height, but rather a contoured surface of an object's physical parameters, such as temperature, chemical reactivity, energy emission in one or more wavelength-specific bands, and more generally the presence or absence of a given physical, chemical, and biological feature.

When the imaged object is sufficiently small and/or simple, its surfaces of measured object parameters may closely follow object shape, although X-ray and other high-resolution microscopy techniques can also detail an object's fundamental molecular and atomic structure at scales below which many physical parameter measurements have no clear meaning. In this wider definition, digital image processing can radically extend the variety, accuracy, and intelligibility of an ever-growing number of scientific and engineering physical measurements.

Context

The use of computers for processing two-dimensional and three-dimensional signals and image data had its most conspicuous origins in the late 1950's through the unmanned planetary science probes, and independently in the Massachusetts Institute of Technology's Geophysical Analysis Group's efforts (later continued by the oil companies) at two- and three-dimensional imaging of the subsurface. Particularly at the Jet Propulsion Laboratory in Pasadena, California, the Surveyor series of space probes returned hundreds of images of the lunar surface, subsequently redisplayed and analyzed in detail to evaluate landing sites for later manned missions.

Although radar and radio astronomy provided the impetus for laying the theoretical groundwork for Fourier analysis and aperture synthesis in the early to mid-1950's, it was particularly following publication of the Cooley-Tukey fast Fourier transform algorithm in the mid-1960's that digital image processing was further developed and applied. An excellent example of digital image processing advanced by both hardware and software developments was the Landsat series of multispectral Earth-orbiting satellite imagery systems.

In the medical sciences, digital image processing was first applied to X-ray images in 1968, and shortly thereafter to automatic classification of genetic chromosomes. More recently, computerized axial tomography permits almost real-time monitoring of human organ functioning.

In oil exploration, the development and implementation of "migration" subsurface imaging techniques, to correct for irregularities in the earth's topography and subsurface velocity, have been responsible for a number of deconvolutional and focusing image-processing techniques. Here, image processing developed as the outcome of two dimensional signal processing.

Many factors in the continuing development of digital image processing include the declining costs and improved capabilities of serial and parallel computer processing and bulk storage devices, as well as increased availability of specialized hardware for image digitizing and graphic display. Many new image-processing developments have accompanied the development of hardware, such as computer array processors specifically designed to process arrays of numerical gray-scale data. The most flexible emergent image-processing systems are interactive computer technologies requiring a minimum of human attention to the underlying operations while allowing rapid human selection and display of the effects of changing processing parameters. The use of artificial intelligence, such as knowledge-based and expert systems, allows the human machine image-processing network to "learn from its past," in turn leading to greater speed and predictability in processing. Expert systems are particularly useful image-processing enhancers when the class of objects imaged represents objects with a number of more or less similar features.

Particularly important in present and future image processing is the continuing development of specialized computer architectures. Because of the two- and three-dimensional nature of massively parallel arrays, so-called multiple subarray architectures significantly speed up parallel-configured signal processors by employing multiple subarrays for simultaneously processing different data streams. In cellular array architectures, all or some of the data flow is extracted from memory and distributed within the array of signal-processing elements. As the total number of memory bits available to each processing element increases, more sophisticated and faster operations and architectures become feasible, and with them more complex and comprehensive image-processing tasks.

Image processing represents a diversified technological interaction between signal processing, artificial intelligence, statistics and probability, and a number of mathematical specialties such as topology and morphometric analysis. Likewise, image processing has found an increasingly wide adaptation and use in almost every branch of the physical, engineering, and natural sciences, as an extension of both measurement and graphic display techniques.

Principal terms

ADAPTIVE: data processing in which processing parameters are varied as measurement of data statistics change

ENHANCEMENT: the improvement of one or more features of an image

INFORMATION: generally, data that have been recorded, classified, and analyzed within a given statistical framework

INTERACTIVE: computer data processing in which the user can modify the processing operations while observing the output results

ITERATIVE: data processing which continues until some condition, such as accuracy or error-tolerance, is satisfied

QUANTIZATION: the restriction of a variable to a discrete number of possible values

RECOVERY: estimating the orientation of an image surface from shades and/or texture

RECURSIVE: refers to processing for which the output depends on previous outputs, as well as the input and intrinsic system response

RESTORATION: an image-processing sequence to correct for effects of known errors and noise

SEGMENTATION: the division of an image into distinct, identifiable regions

Bibliography

Bayes, Gregory A. DIGITAL IMAGE PROCESSING: A PRACTICAL PRIMER. New York: Cascade Press, 1988. One of the more popular and accessible introductory textbooks, stressing practical applications of digital image processing in medicine, physics, astronomy, and several engineering areas. Contains many worthwhile illustrations and flow charts of typical image-processing sequences.

De Graff, C. N., and M. A. Viergever, eds. INFORMATION PROCESSING IN MEDICAL IMAGERY. New York: Plenum Press, 1988. Details the many objects and specific requirements of digital image processing in medical parameter mapping such as thermography, computerized axial tomography, microscopy, and sonography. Includes much incidental material on the historical development of medical imaging, and the relation of medical signal/image processing to advances in computer hardware.

Friedhoff, Richard. VISUALIZATION: THE SECOND COMPUTER REVOLUTION. Salt Lake City, Utah: W. H. Freeman, 1990. Specifically intended for a wider, semitechnical popular audience. Reviews a comprehensive selection of key research developments and applications, containing some firsthand accounts of digital image-processing hardware and software.

Green, William B. DIGITAL IMAGE PROCESSING: A SYSTEMS APPROACH. New York: Van Nostrand Reinhold, 1988. Emphasizes image processing as an extension of one- and two-dimensional signal processing, as specifiable into a number of sequential stages, each corresponding to distinct hardware/software system components. Offers many examples of applications from materials science, physics, and engineering.

Institute of Electrical Engineers. IMAGE PROCESSING AND ITS APPLICATIONS. London: Author, 1989. A grouping of important review and tutorial papers, which together scan the variety of common and less well-known digital image-processing applications. Stresses computer hardware and electrical-engineering aspects such as algorithms; provides some historical information.

Jain, Anil K. FUNDAMENTALS OF DIGITAL IMAGE PROCESSING. Englewood Cliffs, N.J.: Prentice-Hall, 1987. A more advanced text than Bayes, which fully develops one- through three-dimensional Fourier transform and convolutional methods basic to all image processing, as well as a variety of special techniques for enhancing image contrast and coherency. Includes examples from seismology, acoustics and ultrasonics, and tomography.

Levine, M. D. VISION IN MAN AND MACHINE. New York: McGraw-Hill, 1985. Comparable in introductory technical level to Friedhoff's book, cited above. A largely introductory treatment of the more general topic of imaging in human- and machine-based optics. Treats vision as a form of pattern-recognition stages that can be modeled on computers.

Muller, J. P., ed. DIGITAL IMAGE PROCESSING IN REMOTE SENSING. New York: Taylor & Francis, 1988. Focuses on the historical development and applications of digital image processing in satellite imaging of atmospheric, oceanographic, and terrestrial features. A well-organized catalog of some of the earliest image-processing applications.

Russ, J. C. COMPUTER-ASSISTED MICROSCOPY: THE MEASUREMENT AND ANALYSIS OF IMAGES. New York: Plenum Press, 1990. Discusses the requirements and hardware/software technologies available for biomedical and scientific research applications of digital image processing and other image-recording and image-analysis procedures.

Tescher, A. G., ed. APPLICATIONS OF DIGITAL IMAGE PROCESSING. Seattle: SPIE Press, 1989. Considers sonar, radar, manufacturing quality-assurance, and a number of other physics- engineering-oriented uses of digital signal and image processing. Several sections consider image processing in its role as a powerful interdisciplinary methodology.

CT and PET scanners