SPIE Conf. on Machine Vision Systems for Inspection and Metrology VII Copyright July 1998 by F. M. Waltz.

Originally published Boston, Nov. 1998 Duplication by permission only.

Image processing operations in color space using finite-state machines

Frederick M. Waltz 2095 Delaware Avenue, Mendota Heights, MN 55118-4801 USA
ABSTRACT
The SKIPSM (S eparated- K ernel I mage P rocessing using finite-S tate M achines) has recently been extended with excellent results to various grey-scale operations (morphology, Gaussian Blur, ranked filters, etc.) as well as 3-dimensional binary operations. Another interesting direction, initiated in this paper, is the extension to color spaces. After a discussion of some possible goals of such image processing, this paper presents implementation techniques and examples. Because of the extra costs involved with the printing of colored images in journals, the example images are presented here in grey-scale only. Color images will be provided via e-mail on request. Keywords: image color spaces, binary morphology, grey-scale morphology, separability, finite-state machines

1. INTRODUCTION
The motivation for this paper is the opposite of the usual one: Instead of looking for techniques to solve a specified problem, we here investigate possible problems to which a certain technique will apply. To paraphrase a common expression, we have at our disposal an excellent and powerful hammer, and we are looking for things to nail with it. The excellent hammer is the class of image processing implementation techniques called SKIPSM (S eparated-K ernel I mage P rocessing using finite-S tate M achines)1-20, which provides significant speed increases (in comparison to conventional implementations) in a wide range of neighborhood image processing operations. For binary morphology with very large structuring elements and such operations as the Grassfire transform, the speed increases are truly phenomenal, with speedup ratios of 100 or more being common. For other operations, the speedup ratios are typically not as large, but are still worthwhile. Recently, the technique has been extended to a number of grey-scale operations15, 18, 19, 20 and even to 3-dimensional and ndimensional morphology13, with excellent results. This paper explores a few of the possibilities of applying SKIPSM to color images. As a starting point, let us discuss Adobe Photoshop, a widely-used multi-platform software program for manipulating images, and especially color images. A fairly wide range of standard image processing operations is built into this program, including Gaussian and other blurring operations, edge enhancement, nonsharp masking, something they call a median filter, etc. Their standard user documentation does not give their definitions of their operations, but it appears (on the basis of operations investigated so far) that all operations applied to complete color images operate in RGB (Red-Blue-Green) space, with the same grey-scale operation being performed on each color plane. The user can also separate the color planes into three separate images, operate on each plane separately, and then recombine the results to form a new color image. Outputs can be saved in CMYK (Cyan-Magenta-Yellow-Black) format and some other formats, but apparently not in HSI (HueSaturation-Intensity) format, which is the one that generally makes the most sense for industrial inspection. In no case examined to date is there any interaction between the RGB color planes, as there would be if the RGB information was first converted to HSI, these signals were processed separately, and then the results were converted back to RGB. An aside: For experimental purposes (although obviously not for on-line applications), it has proven to be very useful to be able to move images back and forth between Photoshop and a spreadsheet, to allow virtually any kind of image processing operation to be performed without the need to get inside Photoshop or some other image processing engine, or to spend time writing and debugging C code which will the be discarded in a few minutes after your experiment is completed. Techniques for moving images between Photoshop and Excel can be provided on request.

2. SOME POSSIBLE OPERATIONS

As indicated above, conventional color-space operations, on all planes together or on individual planes, include edge enhancement, neighborhood averaging, etc. Figure 1 shows another possibility: Grey-scale erosion or dilation on one of the color planes (RGB or HSI), an operation for which SKIPSM is particularly appropriate15, 19, especially if the desired

Image processing operations in color space using finite-state machines SPIE Paper 3521-33

Frederick M. Waltz

SK20-1

SPIE Conf. on Machine Vision Systems for Inspection and Metrology VII Copyright July 1998 by F. M. Waltz.

Originally published Boston, Nov. 1998 Duplication by permission only.

structuring element is not a simple rectangle or circle. The wavy line in this figure is an attempt to represent the three-dimensional surface of the color variable.

hue saturation or intensity

One possible place where operating on one of the RGB planes might provide useful results is that in which there are image spikes or noise of predominantly one color, say red. Eroding only the red plane would reduce this noise without affecting the other color planes, producing a erosion better signal. Figure 2 presents a synthetic and highly simplified 32x32 example of this type, in which 3x3 spikes in the red plane are completely y image plane x removed by eroding with a 5x5 circular structuring element (shown below), with only minor changes a 2-pixel enlargement of the central dark square in the main structure in this plane. (Because color plates are Figure 1. Grey-scale erosion on individual not practical in this Proceedings volume, grey-scale versions of the color-space variables. before and after images are shown here. These contain only the intensity information (the I plane of the HSI representation). The smallest discernible squares in these images represent one pixel. Color versions of these and the other images from this article will be provided on request via e-mail. Contact the author at <WaltzMN@PioneerPlanet.infi.net>. Similarly, dropouts or dark spots can be reduced Red After Red Before by grey-scale dilation on one or more of the color planes. Figure 3 shows grey-scale representations of a color image before and after the application of a diamond-shaped 7x7 dilation operator (shown at right) to the green plane only.

Green

Blue

Figure 3. A simple 32x32 synthetic image before and after dilation of the green color plane with a 7x7 diamond. There has, of course, been some distortion of the signal aspect of the image, in that the central light area of the green plane of the image has been enlarged by three pixels all around. Except in very unusual situations, a filtering operation distorts the signal while removing the noise. The point being made here is that if additional kinds of image processing operations are available, there is a better chance that one can be found to reduce the noise effectively without distorting the signal too much. These are clearly highly artificial examples. Should larger erosion or dilation operators be desired, SKIPSM can provide efficient implementations, including automatically-generated compileable main-loop computer code19.
Frederick M. Waltz SK20-2

Grey Before

Grey After

Figure 2. A simple 32x32 synthetic image before and after erosion of the red color plane with a 5x5 circle.

Image processing operations in color space using finite-state machines SPIE Paper 3521-33

SPIE Conf. on Machine Vision Systems for Inspection and Metrology VII Copyright July 1998 by F. M. Waltz.

Originally published Boston, Nov. 1998 Duplication by permission only.

3. USING DIFFERENT OPERATIONS SIMULTANEOUSLY ON THE RGB COLOR SIGNALS

Figure 4 shows another possibility: Applying the same operation (erosion, in this example) to all the color signals or planes, but using different-sized structuring elements. This would be appropriate for RGB images if the noise to be filtered out had different size and shapes in the color planes note likely, but certainly possible, depending on the source of the noise. For hue example, varying amounts of light in the red band might be affecting the (or R) acquired image adversely. Varying light intensity and color has been a real problem in outdoor automated vehicle recognition and control systems on high-speed highways, because the ambient light is much more red near dawn and sunset. (Human eyes readily adjust to this, so that humans are erosion y generally unaware of slow ambient light changes, but electronic cameras x saturation usually dont handle this problem very well.) (or G) Rather than continue with more morphology examples at this point, a linear processing technique will now be considered. A standard method of finding particular features in an image, such as holes or corners, is to use y linear filters designed to trap the features in question. These are called n-tuples in the PIP image processing environment16, but are not intensity x (or B) erosion available in typical image processing designed to produce images for human viewing. The n-tuple operation involves computing the correlation of pixels in the image with the selected (usually non-contiguous) pixels of the n-tuple. SKIPSM can implement this operation rather well. y erosion In the example shown in Figure 5, three x image plane independent red green blue n-tuple spot lower upper Figure 4. Using different operations on detectors are detector detector detector the individual color-space variables. applied to the RGB planes. In the red plane, the detector looks for bright spots surrounded by a darker area (pupil of eye). In the green Key: positive weight plane, the detector looks for a light area below a darker area negative weight a) (lower eyelid). In the blue plane, the detector looks for a light center pixel area above a darker area (upper eyelid). The combined greyscale intensity image (Figure 5e), because it merges the three fairly good individual detectors, results in not very good detection of any of the three entities, showing one of the advantages of using separate b) c) d) color planes for multiple detection tasks. Note that Figure 5. a) 32x32 image of an eye. b) pupil detector - red. because the center spot also c) lower eyelid detector - green. d) upper eyelid detector meets the other two definitions, blue. e) combined detectors - intensity only. it provides a strong response on all three planes. Larger n-tuples could be used to eliminate these ambiguities. Another interesting possibility for which SKIPSM is well adapted: grey-scale openings and closings. Figure 6 shows a case in which grey-scale erosion of one or more of the color variable is followed by grey-scale dilation of the result. Figure 7a) shows the application of this sequence using a 5x5 circular structuring element to all three color planes (RGB) of the
Image processing operations in color space using finite-state machines SPIE Paper 3521-33 Frederick M. Waltz

e)
SK20-3

SPIE Conf. on Machine Vision Systems for Inspection and Metrology VII Copyright July 1998 by F. M. Waltz.

Originally published Boston, Nov. 1998 Duplication by permission only.

eye image of Figure 5a). The resulting RGB results and the grey-scale intensity-only information are shown in Figure 7 a real black eye. Figure 8 shows the RGB images and the resulting grey-scale intensity-only image resulting from subtracting these color images from the original color image. Note the way that the pupil has been highlighted.
Red Green

dilation

intensity etc.

dilated contour

original contour eroded contour second operation

y intensity etc.

x original contour eroded contour erosion first operation x

image plane

Blue

Intensity

Figure 6. Grey-scale erosion followed by grey-scale dilation of an individual color variable.

4. PROCESSING IN HSI SPACE

All the actual examples (as opposed to the discussion) in this paper have involved processing in RGB space. This was done primarily because inputs and outputs in RGB format (but apparently not HSI) are available in Photoshop, the tool used for preparing and publishing this paper. (Photoshop is Figure 7. Result of applying a 3x3 erosion diamond followed primarily designed for the publishing business.) Thus, even by 3x3 dilation diamond to Figure 5a). though the actual image processing shown here was done in Excel, which can handle any kind of computation that one Green Red could desire to do, there was insufficient time to make the complicated nonlinear conversions to and from HSI space. Therefore, such examples are not provided here. The fact remains, however, that for many image processing applications, operations in HSI space can provide significant advantages. Of course, operation in Intensity space is familiar to all, because this is nothing other than the conventional grey-scale processing used in most machine vision system. For example, a line or feature on an object of one predominant color but varying intensity and purity will appear as a highly variable signal in two or perhaps all three of the RGB planes, as well as the S (saturation) and I (intensity) planes in HSI space. But in H (hue) space, it will show us as a contiguous set of pixels with a small color range, making detection and location of the feature relatively easy.

Blue

Intensity

Because the goal of this paper is to present possible uses of the kind of processing SKIPSM can provide, no more will be said here on this subject. The author expects to explore Figure 8. Result of subtracting Figure 7 from original image further the implications of SKIPSM for automated inspection shown in Figure 5a) to highlight things removed. using color images in HSI as well as RGB space.
Image processing operations in color space using finite-state machines SPIE Paper 3521-33 Frederick M. Waltz SK20-4

SPIE Conf. on Machine Vision Systems for Inspection and Metrology VII Copyright July 1998 by F. M. Waltz.

Originally published Boston, Nov. 1998 Duplication by permission only.

5 . SUMMARY AND CONCLUSIONS

This paper has presented ideas for using the high-speed and/or large-neighborhood capabilities of the SKIPSM paradigm to facilitate automated inspection involving colored images. The emphasis has been on presenting possibilities for processing, not on the advantages of one or another color inspection approach. This is a large field in its own right. It is hoped that experts in this field will find reasons here to investigate SKIPSM implementations for operations that might otherwise be too time consuming to be practical for on-line inspection.

6. BIBLIOGRAPHY
1. F. M. Waltz, SKIPSM: Separated-Kernel Image Processing using finite-State Machines, Proc. SPIE Conf. on Machine Vision Applications, Architectures, and Systems Integration III, Vol. 2347, Paper No. 36, Boston, Nov. 1994 2. F. M. Waltz and H. H. Garnaoui, Application of SKIPSM to binary morphology, Proc. SPIE Conf. on Machine Vision Applications, Architectures, and Systems Integration III, Vol. 2347, Paper No. 37, Boston, Nov. 1994 3. F. M. Waltz and H. H. Garnaoui, Fast computation of the Grassfire Transform using SKIPSM, Proc. SPIE Conf. on Machine Vision Applications, Architectures, and Systems Integration III, Vol. 2347, Paper No. 38, Boston, Nov. 1994 4. F. M. Waltz, Application of SKIPSM to binary template matching, Proc. SPIE Conf. on Machine Vision Applications, Architectures, and Systems Integration III, Vol. 2347, Paper No. 39, Boston, Nov. 1994 5. F. M. Waltz, Application of SKIPSM to grey-level morphology, Proc. SPIE Conf. on Machine Vision Applications, Architectures, and Systems Integration III, Vol. 2347, Paper No. 40, Boston, Nov. 1994 6. F. M. Waltz, Application of SKIPSM to the pipelining of certain global image processing operations, Proc. SPIE Conf. on Machine Vision Applications, Architectures, and Systems Integration III, Vol. 2347, Paper No. 41, Boston, Nov. 1994 7. A. A. Hujanen & F. M. Waltz, Pipelined implementation of binary skeletonization using finite-state machines, Proc. SPIE Conf. on Machine Vision Applications in Industrial Inspection, Vol. 2423, Paper No. 2, San Jose, Feb. 1995 8. A. A. Hujanen & F. M. Waltz, Extending the SKIPSM binary skeletonization implementation, Proc. SPIE Conf. on Machine Vision Applications, Architectures, and Systems Integration IV, Vol. 2597, Paper No. 12, Philadelphia, Oct. 1995 9. F. M. Waltz, Application of SKIPSM to binary correlation, Proc. SPIE Conf. on Machine Vision Applications, Architectures, and Systems Integration IV, Vol. 2597, Paper No. 11, Philadelphia, Oct. 1995 10. F. M. Waltz, SKIPSM implementations: morphology and much, much more, Proc. SPIE Conf. on Machine Vision Applications, Architectures, and Systems Integration IV, Vol. 2597, Paper No. 14, Philadelphia, Oct. 1995 11. F. M. Waltz, Automated generation of finite-state machine lookup tables for binary morphology, Proc. SPIE Conf. on Machine Vision Applications, Architectures, and Systems Integration V, Boston, Nov. 1996 12. F. M. Waltz, Binary openings and closings in one pass using finite-state machines, Proc. SPIE Conf. on Machine Vision Applications, Architectures, and Systems Integration V, Boston, Nov. 1996 13. F. M. Waltz, Implementation of SKIPSM for 3-D binary morphology,Proc. SPIE Conf. on Machine Vision Applications, Architectures, and Systems Integration VI, Vol. 3205, Paper No. 13, Pittsburgh, Oct. 1997 14. F. M. Waltz, Binary dilation using SKIPSM: Some interesting variations, Proc. SPIE Conf. on Machine Vision Applications, Architectures, and Systems Integration VI, Vol. 3205, Paper No. 15, Pittsburgh, Oct. 1997 15. J. W. V. Miller and F. M. Waltz, Software implementation of 2-D grey-level dilation using SKIPSM, F. M. Waltz and J. W. V. Miller, An efficient algorithm for Gaussian blur using finite-state machines, Proc. SPIE Conf. on Machine Vision Systems for Inspection and Metrology VII, Vol. 3521, Paper No. 18, Pittsburgh, Oct. 1997

Image processing operations in color space using finite-state machines SPIE Paper 3521-33

Frederick M. Waltz

SK20-5

SPIE Conf. on Machine Vision Systems for Inspection and Metrology VII Copyright July 1998 by F. M. Waltz.

Originally published Boston, Nov. 1998 Duplication by permission only.

16. R. Hack, F. M. Waltz, & B. G. Batchelor, Software implementation of the SKIPSM paradigm under PIP, Proc. SPIE Conf. on Machine Vision Applications, Architectures, and Systems Integration VI, Vol. 3205, Paper No. 19, Pittsburgh, Oct. 1997 17. F. M. Waltz, The application of SKIPSM to various 3x3 image processing operations, Proc. SPIE Conf. on Machine Vision Systems for Inspection and Metrology VII, Vol. 3521, Paper No. 30, Boston, Nov. 1998 18. F. M. Waltz, R. Hack, and B. G. Batchelor, Fast, efficient algorithms for 3x3 ranked filters using finite-state machines, Proc. SPIE Conf. on Machine Vision Systems for Inspection and Metrology VII, Vol. 3521, Paper No. 31, Boston, Nov. 1998 19. F. M. Waltz, Automated generation of efficient code for grey-scale image processing, Proc. SPIE Conf. on Machine Vision Systems for Inspection and Metrology VII, Vol. 3521, Paper No. 32, Boston, Nov. 1998 20. F. M. Waltz and J. W. V. Miller, An efficient algorithm for Gaussian blur using finite-state machines, Proc. SPIE Conf. on Machine Vision Systems for Inspection and Metrology VII, Vol. 3521, Paper No. 37, Boston, Nov. 1998

Image processing operations in color space using finite-state machines SPIE Paper 3521-33

Frederick M. Waltz

SK20-6