This assignment focuses on methods of skin detection based on various algorithms for determining whether or not pixels are skin colored. I have based my tests for skin pixels on a combination of hue, saturation, intensity, and RGB values at individual pixels.

The main program I have developed for this assignment takes as input a .ppm or .pnm ASCII color image. It outputs two images which are based on the original. The first is a copy of the input with all of the pixels determined to be "skin pixels" marked in black. The second is a copy of the input with the determined "center" of the skin marked. The mode of the skin values (i.e. the row and column with more skin pixels than any other row or column respectively) is marked with a red cross. The mean of the skin values (i.e. the row and column which are the mathematical averages of the rows and columns where all skin pixels are located) is marked with a green cross.

My goal has been to determine the best combination of color factors to differentiate skin pixels from non-skin pixels on the three input images I have been using in my tests. I have tried to have as many skin pixels as possible identified correctly, while keeping the number of non-skin pixels identified incorrectly as skin pixels to a minimum. In doing so, I have also tried to find a method of determining the "center" of an area of pixels determined to be skin.

Clearly it would be impossible to find a precise definition of "skin" based on attributes of color alone which would correctly identify skin pixels and non-skin pixels in all cases. The color of skin can vary greatly from one person to the next. In addition, the background lighting in a picture can have a huge effect on what color the skin appears to be.

Because of these factors, my program is not designed to work well on all input pictures. I have used three main images in my testing. One is an image of professor Betke, betke.ppm. Two are images of myself, jenn.pnm and jenn2.pnm. The lighting in these two pictures is different, and as a result the hue of my skin is not the same. My program was designed to work well on betke.pnm and jenn.pnm. I have included the results on jenn2.pnm in order to illustrate the extreme difference which lighting can make.

Since these three images were the main focus of my research, I realize that my skin tests are somewhat biased towards them. I do not claim that these tests will work well on other images.

One further note I should make is that all .jpg images which are referenced from this page were converted directly from .ppm and .pnm images using the GIMP, and have not been altered in any other way. I have convered them to .jpg images so that they will display on this page. The .ppm and .pnm versions of these images are also included in my project tar file. The original images of Professor Betke and myself are in the origimages directory. The skin sample images are in the data directory. The images output by my program are in the outfiles directory.

In order to find a starting point for my experiments, I created a program skintest.cpp which is analyzes the pixels in an image and outputs information about the colors used in the image. I then ran this program on three skin images, skin.ppm which is a patch of skin from betke.ppm, skin2.pnm which is a patch of skin from jenn.pnm, and skin3.pnm which is a patch of skin from jenn2.pnm. I planned on using information collected from these skin samples to determine my tests for what constitutes a skin pixel.

My first version of skintest.cpp looked at nothing but the red, green, and blue values of pixels. It output the average red, green, and blue values over the image. These values, taken seperately, were not similar enough in the three pictures to determine anything. However, I did notice that the ratios of the average red values to the average green values in the three pictures were close. To find out how close, I added the red to green ratio to the output of skintest.cpp. The results were as follows, with the color values ranging from 0 to 255.

Image:	Average Red Value:	Average Green Value:	Average Blue Value:	Per Pixel Ave Red/Green:	Combined Ave Red/Green:
skin1.ppm	170.205882	146.421569	115.722222	1.162028	1.162437
skin2.pnm	214.843175	183.222857	175.265397	1.174104	1.172578
skin3.pnm	133.149767	105.522145	134.152681	1.261755	1.261818

Based on these results, I made my first attempt at skin detection. This version of the skin test checked the ratio of red to green in a pixel to see if it fell into a certain range. The range I had the best luck with on the particular images I used was between 1.1 and 1.3. However, even with the best range I could find, this method was lacking. Many real skin pixels were marked as skin pixels, but a good amount were missed, and a fair amount of background pixels were also marked as skin pixels. (The result of this attempt and the others can be viewed in the Image Results section below.)

Here is the facePixel function used in my implementation of the Red/Green Ratio Method.

This ratio seemed like a good starting point even though I knew that on its own, it was not enough, so I decided to move further in the same direction by checking the hue, saturation, and/or intensity values of pixels. To convert to an HSI system, I used the procedure provided in class by Professor Betke. In order to figure out the appropriate range of HSI values, I added average hue, saturation, and intensity to the output of skintest. These were the results on the same three patches of skin used before. The average HSI values over the skin samples were as follows.

Image:	Average Hue:	Average Saturation:	Average Intensity:
skin1.ppm	5.998102	0.321071	170.205882
skin2.pnm	6.021008	0.186734	214.843175
skin3.pnm	3.838065	0.234735	138.148601

For my second attempt at skin detection, I defined a span of values which would pick up most of the skin pixels from the first two images and some from the third. I started with estimates from the data I had collected. After some experimentation, the spans I found to work the best were

5.75 <= hue <= 6.30
0.10 <= sat <= 0.40
100 <= int <= 250

Using these ranges, my program could correctly identify the majority of skin pixels but still incorrectly marked too many background pixels as skin pixels to be effective on its own.

Here is the facePixel function used in my implementation of the HSI Method.

I felt that my first two attempts were leading me in the right direction. However, both attempts labeled too many background pixels as skin pixels. I based my third attempt at identifying skin pixels on a combination of these two methods. I hoped that more background pixels would be weeded out if skin pixels were required to pass tests on their red to green ratio in addition to their hue, saturation, and intensity. Again I experimented with the ranges I used. I found the best to be

5.70 <= hue <= 6.50
0.10 <= saturation <= 0.40
100 <= intensity <= 250
1.0 < red/green < 1.3

Here is the facePixel function used in my implementation of the Combination Red/Green Ratio and HSI Method.

I was satisfied with the results of this final method on the first two images. However, it worked poorly on the third image. These results will be discussed in more detail below.

The following are the source code files which implement the face detection methods as described above. These are the most recent versions of the files, with the Combination Red/Green and HSI facePixel function in place.

vision.h -- This is the header file for vision.cpp and main.cpp. It is based on the header file provided by Margrit Betke for use with the assignment. Some changes have been made to expand it to cover main.cpp and the conversions to HSI color values.

vision.cpp -- This file was taken virtually as-is from a sample file provided by and written by Margrit Betke for Image and Video Computing. It contains functions for manipulating .ppm files. The functions included can scan and write .ppm files. In addition I have added some functionality for converting RGB values of pixels to HSI values.

main.cpp -- This is the main program of my face detection project. Two output files will be produced when this program is run. The first will display the image with the face pixels in black. The second will display the image with the center of the face marked in two ways. The mode of the face pixels will be marked with a red cross, the mean with a green cross.

In addition, the Makefile I used to compile the main program is included in the code directory. I compiled everything in Mandrake Linux 8.1. I believe that the Makefile should work on any unix platform.

Here are the three original images I used in my tests as well as the results of the three skin detection methods used above on each of the images.


Image 1:

The Original:	Red/Green Method:
HSI Method:	Combination of Red/Green and HSI Methods:

Image 2:

(These images are displayed smaller than they actually are to fit on the page. Click on an image to see the full version.)

The Original:

Red/Green Method:

HSI Method:

Combination of Red/Green and HSI Methods:

Image 3:

(Again, these images are displayed smaller than they actually are to fit on the page. Click on an image to see the full version.)

The Original:

Red/Green Method:

HSI Method:

Combination of Red/Green and HSI Methods:

The Red/Green Ratio Method proved to be ineffective by itself for determining skin pixels. Although it found some of the skin pixels in Image 1, it missed the area above Professor Betke's eyes, and all of her neck. In Image 2, it picked up most of the skin pixels but also incorrectly identified features in the background such as the border of the poster and parts of the printer as skin. It is interesting to note that this method performed best of the three methods I tried on Image 3. However, as mentioned in my Assumptions, I was looking for the method which would perform best on the first two images.

The HSI Method picked up almost all of the skin pixels in Image 1. In addition, it correctly excluded areas on Professor Betke's neck where her necklace was. Unfortunately, this method identified the entire lighter portion of the background and Professor Betke's shirt as skin. Clearly this method is not precise enough to be useful on this particular image. The method did perform well on Image 2. Almost all of the skin was identified, and much less of the background was identified incorrectly than with the first method. The method performed poorly on Image 3, missing most of the skin entirely as well as identifying part of the background as skin.

The Red/Green and HSI Combination Method clearly performed the best on Image 1. The output from Image 1 is quite satisfying. Almost all of the face skin is correctly identified and the eyes are propperly excluded. However, this method still included Professor Betke's teeth and a small amount of her hair as skin. This method performed reasonably well on Image 2 also, although not as well as method 2. More skin pixels have been left out and there is still a small ammount of background noise which has been marked as skin. Like method 2, this method did particularly poorly on Image 3, although less of the background is identified incorrectly this time.

The Combination Method seems to determine the "center" of the skin areas best in each image as well. In Images 1 and 2, the mean center and the mode center as specified by this Method are more or less accurate representations of the centers of the skin areas. In Image 3, the mode center is off due to the large ammount of background which has been identified as skin. However, the mean center is still fairly accurate.

My overall conclusion is that skin detection algorithms should not be based on the color of pixels alone. There are too many outside factors such as lighting that can change the apparent color of skin, and of course, different people have different colored skin. In addition, objects in the background may be the same color as a person's skin and there is no clear way of telling the difference using these methods.

However, if these methods must be used, it is wise to use a combination of color-related factors. The red, green, and blue values of a pixel are not enough. By adding hue, saturation, and intensity, a more defined idea of skin color can be formed.

All content on this page was written by Jennifer Wortman in January 2002, except where noted.